API tools faq
paste
Advertisement
ERENARD63

Building your Recurrent Neural Network - Step by Step

ERENARD63
May 19th, 2018
549
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 45.74 KB | None | 0 0
raw download clone embed print report
  1.  
  2. # coding: utf-8
  3.  
  4. # # Building your Recurrent Neural Network - Step by Step
  5. #
  6. # Welcome to Course 5's first assignment! In this assignment, you will implement your first Recurrent Neural Network in numpy.
  7. #
  8. # Recurrent Neural Networks (RNN) are very effective for Natural Language Processing and other sequence tasks because they have "memory". They can read inputs $x^{\langle t \rangle}$ (such as words) one at a time, and remember some information/context through the hidden layer activations that get passed from one time-step to the next. This allows a uni-directional RNN to take information from the past to process later inputs. A bidirection RNN can take context from both the past and the future.
  9. #
  10. # **Notation**:
  11. # - Superscript $[l]$ denotes an object associated with the $l^{th}$ layer.
  12. #     - Example: $a^{[4]}$ is the $4^{th}$ layer activation. $W^{[5]}$ and $b^{[5]}$ are the $5^{th}$ layer parameters.
  13. #
  14. # - Superscript $(i)$ denotes an object associated with the $i^{th}$ example.
  15. #     - Example: $x^{(i)}$ is the $i^{th}$ training example input.
  16. #
  17. # - Superscript $\langle t \rangle$ denotes an object at the $t^{th}$ time-step.
  18. #     - Example: $x^{\langle t \rangle}$ is the input x at the $t^{th}$ time-step. $x^{(i)\langle t \rangle}$ is the input at the $t^{th}$ timestep of example $i$.
  19. #    
  20. # - Lowerscript $i$ denotes the $i^{th}$ entry of a vector.
  21. #     - Example: $a^{[l]}_i$ denotes the $i^{th}$ entry of the activations in layer $l$.
  22. #
  23. # We assume that you are already familiar with `numpy` and/or have completed the previous courses of the specialization. Let's get started!
  24.  
  25. # Let's first import all the packages that you will need during this assignment.
  26.  
  27. # In[1]:
  28.  
  29. import numpy as np
  30. from rnn_utils import *
  31.  
  32.  
  33. # ## 1 - Forward propagation for the basic Recurrent Neural Network
  34. #
  35. # Later this week, you will generate music using an RNN. The basic RNN that you will implement has the structure below. In this example, $T_x = T_y$.
  36.  
  37. # Here's how you can implement an RNN:
  38. #
  39. # **Steps**:
  40. # 1. Implement the calculations needed for one time-step of the RNN.
  41. # 2. Implement a loop over $T_x$ time-steps in order to process all the inputs, one at a time.
  42. #
  43. # Let's go!
  44. #
  45. # ## 1.1 - RNN cell
  46. #
  47. # A Recurrent neural network can be seen as the repetition of a single cell. You are first going to implement the computations for a single time-step. The following figure describes the operations for a single time-step of an RNN cell.
  48. #
  49. # <img src="images/rnn_step_forward.png" style="width:700px;height:300px;">
  50. # <caption><center> **Figure 2**: Basic RNN cell. Takes as input $x^{\langle t \rangle}$ (current input) and $a^{\langle t - 1\rangle}$ (previous hidden state containing information from the past), and outputs $a^{\langle t \rangle}$ which is given to the next RNN cell and also used to predict $y^{\langle t \rangle}$ </center></caption>
  51. #
  52. # **Exercise**: Implement the RNN-cell described in Figure (2).
  53. #
  54. # **Instructions**:
  55. # 1. Compute the hidden state with tanh activation: $a^{\langle t \rangle} = \tanh(W_{aa} a^{\langle t-1 \rangle} + W_{ax} x^{\langle t \rangle} + b_a)$.
  56. # 2. Using your new hidden state $a^{\langle t \rangle}$, compute the prediction $\hat{y}^{\langle t \rangle} = softmax(W_{ya} a^{\langle t \rangle} + b_y)$. We provided you a function: `softmax`.
  57. # 3. Store $(a^{\langle t \rangle}, a^{\langle t-1 \rangle}, x^{\langle t \rangle}, parameters)$ in cache
  58. # 4. Return $a^{\langle t \rangle}$ , $y^{\langle t \rangle}$ and cache
  59. #
  60. # We will vectorize over $m$ examples. Thus, $x^{\langle t \rangle}$ will have dimension $(n_x,m)$, and $a^{\langle t \rangle}$ will have dimension $(n_a,m)$.
  61.  
  62. # In[2]:
  63.  
  64. # GRADED FUNCTION: rnn_cell_forward
  65.  
  66. def rnn_cell_forward(xt, a_prev, parameters):
  67.     """
  68.    Implements a single forward step of the RNN-cell as described in Figure (2)
  69.  
  70.    Arguments:
  71.    xt -- your input data at timestep "t", numpy array of shape (n_x, m).
  72.    a_prev -- Hidden state at timestep "t-1", numpy array of shape (n_a, m)
  73.    parameters -- python dictionary containing:
  74.                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)
  75.                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)
  76.                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
  77.                        ba --  Bias, numpy array of shape (n_a, 1)
  78.                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
  79.    Returns:
  80.    a_next -- next hidden state, of shape (n_a, m)
  81.    yt_pred -- prediction at timestep "t", numpy array of shape (n_y, m)
  82.    cache -- tuple of values needed for the backward pass, contains (a_next, a_prev, xt, parameters)
  83.    """
  84.    
  85.     # Retrieve parameters from "parameters"
  86.     Wax = parameters["Wax"]
  87.     Waa = parameters["Waa"]
  88.     Wya = parameters["Wya"]
  89.     ba = parameters["ba"]
  90.     by = parameters["by"]
  91.    
  92.     ### START CODE HERE ### (≈2 lines)
  93.     # compute next activation state using the formula given above
  94.     #print("a_prev",np.shape(a_prev))
  95.     #print("xt",np.shape(xt))
  96.     a_next = np.tanh(np.dot(Waa,a_prev)+np.dot(Wax,xt)+ba)
  97.     #np.multiply(Waa,a_prev)+
  98.     # compute output of the current cell using the formula given above
  99.     yt_pred = softmax(np.dot(Wya,a_next)+by)  
  100.     ### END CODE HERE ###
  101.    
  102.     # store values you need for backward propagation in cache
  103.     cache = (a_next, a_prev, xt, parameters)
  104.    
  105.     return a_next, yt_pred, cache
  106.  
  107.  
  108. # In[3]:
  109.  
  110. np.random.seed(1)
  111. xt = np.random.randn(3,10)
  112. a_prev = np.random.randn(5,10)
  113. Waa = np.random.randn(5,5)
  114. Wax = np.random.randn(5,3)
  115. Wya = np.random.randn(2,5)
  116. ba = np.random.randn(5,1)
  117. by = np.random.randn(2,1)
  118. parameters = {"Waa": Waa, "Wax": Wax, "Wya": Wya, "ba": ba, "by": by}
  119.  
  120. a_next, yt_pred, cache = rnn_cell_forward(xt, a_prev, parameters)
  121. print("a_next[4] = ", a_next[4])
  122. print("a_next.shape = ", a_next.shape)
  123. print("yt_pred[1] =", yt_pred[1])
  124. print("yt_pred.shape = ", yt_pred.shape)
  125.  
  126.  
  127.  
  128. # ## 1.2 - RNN forward pass
  129. #
  130. # You can see an RNN as the repetition of the cell you've just built. If your input sequence of data is carried over 10 time steps, then you will copy the RNN cell 10 times. Each cell takes as input the hidden state from the previous cell ($a^{\langle t-1 \rangle}$) and the current time-step's input data ($x^{\langle t \rangle}$). It outputs a hidden state ($a^{\langle t \rangle}$) and a prediction ($y^{\langle t \rangle}$) for this time-step.
  131. #
  132. #
  133. # <img src="images/rnn.png" style="width:800px;height:300px;">
  134. # <caption><center> **Figure 3**: Basic RNN. The input sequence $x = (x^{\langle 1 \rangle}, x^{\langle 2 \rangle}, ..., x^{\langle T_x \rangle})$  is carried over $T_x$ time steps. The network outputs $y = (y^{\langle 1 \rangle}, y^{\langle 2 \rangle}, ..., y^{\langle T_x \rangle})$. </center></caption>
  135. #
  136. #
  137. #
  138. # **Exercise**: Code the forward propagation of the RNN described in Figure (3).
  139. #
  140. # **Instructions**:
  141. # 1. Create a vector of zeros ($a$) that will store all the hidden states computed by the RNN.
  142. # 2. Initialize the "next" hidden state as $a_0$ (initial hidden state).
  143. # 3. Start looping over each time step, your incremental index is $t$ :
  144. #     - Update the "next" hidden state and the cache by running `rnn_cell_forward`
  145. #     - Store the "next" hidden state in $a$ ($t^{th}$ position)
  146. #     - Store the prediction in y
  147. #     - Add the cache to the list of caches
  148. # 4. Return $a$, $y$ and caches
  149.  
  150. # In[4]:
  151.  
  152. # GRADED FUNCTION: rnn_forward
  153.  
  154. def rnn_forward(x, a0, parameters):
  155.     """
  156.    Implement the forward propagation of the recurrent neural network described in Figure (3).
  157.  
  158.    Arguments:
  159.    x -- Input data for every time-step, of shape (n_x, m, T_x).
  160.    a0 -- Initial hidden state, of shape (n_a, m)
  161.    parameters -- python dictionary containing:
  162.                        Waa -- Weight matrix multiplying the hidden state, numpy array of shape (n_a, n_a)
  163.                        Wax -- Weight matrix multiplying the input, numpy array of shape (n_a, n_x)
  164.                        Wya -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
  165.                        ba --  Bias numpy array of shape (n_a, 1)
  166.                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
  167.  
  168.    Returns:
  169.    a -- Hidden states for every time-step, numpy array of shape (n_a, m, T_x)
  170.    y_pred -- Predictions for every time-step, numpy array of shape (n_y, m, T_x)
  171.    caches -- tuple of values needed for the backward pass, contains (list of caches, x)
  172.    """
  173.    
  174.     # Initialize "caches" which will contain the list of all caches
  175.     caches = [a0,x]
  176.    
  177.     # Retrieve dimensions from shapes of x and parameters["Wya"]
  178.     n_x, m, T_x = x.shape
  179.     n_y, n_a = parameters["Wya"].shape
  180.    
  181.     ### START CODE HERE ###
  182.    
  183.     # initialize "a" and "y" with zeros (≈2 lines)
  184.     a = np.zeros((n_a, m,T_x))
  185.     y_pred = np.zeros((n_y, m, T_x))
  186.    
  187.     # Initialize a_next (≈1 line)
  188.     a_next = a0
  189.    
  190.     # loop over all time-steps
  191.    
  192.     for t in range(T_x):
  193.         # Update next hidden state, compute the prediction, get the cache (≈1 line)
  194.         #print("x", np.shape(x))
  195.         #print("a_next",a_next)
  196.         a_next, yt_pred, cache = rnn_cell_forward(x[:,:,t], a_next, parameters)
  197.         # Save the value of the new "next" hidden state in a (≈1 line)
  198.        
  199.         a[:,:,t] = a_next
  200.         # Save the value of the prediction in y (≈1 line)
  201.         y_pred[:,:,t] = yt_pred
  202.         # Append "cache" to "caches" (≈1 line)
  203.         caches=(caches,cache)
  204.        
  205.     ### END CODE HERE ###
  206.    
  207.     # store values needed for backward propagation in cache
  208.     caches = (caches, x)
  209.    
  210.     return a, y_pred, caches
  211.  
  212.  
  213. # In[5]:
  214.  
  215. np.random.seed(1)
  216. x = np.random.randn(3,10,4)
  217. a0 = np.random.randn(5,10)
  218. Waa = np.random.randn(5,5)
  219. Wax = np.random.randn(5,3)
  220. Wya = np.random.randn(2,5)
  221. ba = np.random.randn(5,1)
  222. by = np.random.randn(2,1)
  223. parameters = {"Waa": Waa, "Wax": Wax, "Wya": Wya, "ba": ba, "by": by}
  224.  
  225. a, y_pred, caches = rnn_forward(x, a0, parameters)
  226. print("a[4][1] = ", a[4][1])
  227. print("a.shape = ", a.shape)
  228. print("y_pred[1][3] =", y_pred[1][3])
  229. print("y_pred.shape = ", y_pred.shape)
  230. print("caches[1][1][3] =", caches[1][1][3])
  231. print("len(caches) = ", len(caches))
  232.  
  233.  
  234.  
  235. # Congratulations! You've successfully built the forward propagation of a recurrent neural network from scratch. This will work well enough for some applications, but it suffers from vanishing gradient problems. So it works best when each output $y^{\langle t \rangle}$ can be estimated using mainly "local" context (meaning information from inputs $x^{\langle t' \rangle}$ where $t'$ is not too far from $t$).
  236. #
  237. # In the next part, you will build a more complex LSTM model, which is better at addressing vanishing gradients. The LSTM will be better able to remember a piece of information and keep it saved for many timesteps.
  238.  
  239. # ## 2 - Long Short-Term Memory (LSTM) network
  240. #
  241. # This following figure shows the operations of an LSTM-cell.
  242. #
  243. # <img src="images/LSTM.png" style="width:500;height:400px;">
  244. # <caption><center> **Figure 4**: LSTM-cell. This tracks and updates a "cell state" or memory variable $c^{\langle t \rangle}$ at every time-step, which can be different from $a^{\langle t \rangle}$. </center></caption>
  245. #
  246. # Similar to the RNN example above, you will start by implementing the LSTM cell for a single time-step. Then you can iteratively call it from inside a for-loop to have it process an input with $T_x$ time-steps.
  247. #
  248. # ### About the gates
  249. #
  250. # #### - Forget gate
  251. #
  252. # For the sake of this illustration, lets assume we are reading words in a piece of text, and want use an LSTM to keep track of grammatical structures, such as whether the subject is singular or plural. If the subject changes from a singular word to a plural word, we need to find a way to get rid of our previously stored memory value of the singular/plural state. In an LSTM, the forget gate lets us do this:
  253. #
  254. # $$\Gamma_f^{\langle t \rangle} = \sigma(W_f[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_f)\tag{1} $$
  255. #
  256. # Here, $W_f$ are weights that govern the forget gate's behavior. We concatenate $[a^{\langle t-1 \rangle}, x^{\langle t \rangle}]$ and multiply by $W_f$. The equation above results in a vector $\Gamma_f^{\langle t \rangle}$ with values between 0 and 1. This forget gate vector will be multiplied element-wise by the previous cell state $c^{\langle t-1 \rangle}$. So if one of the values of $\Gamma_f^{\langle t \rangle}$ is 0 (or close to 0) then it means that the LSTM should remove that piece of information (e.g. the singular subject) in the corresponding component of $c^{\langle t-1 \rangle}$. If one of the values is 1, then it will keep the information.
  257. #
  258. # #### - Update gate
  259. #
  260. # Once we forget that the subject being discussed is singular, we need to find a way to update it to reflect that the new subject is now plural. Here is the formulat for the update gate:
  261. #
  262. # $$\Gamma_u^{\langle t \rangle} = \sigma(W_u[a^{\langle t-1 \rangle}, x^{\{t\}}] + b_u)\tag{2} $$
  263. #
  264. # Similar to the forget gate, here $\Gamma_u^{\langle t \rangle}$ is again a vector of values between 0 and 1. This will be multiplied element-wise with $\tilde{c}^{\langle t \rangle}$, in order to compute $c^{\langle t \rangle}$.
  265. #
  266. # #### - Updating the cell
  267. #
  268. # To update the new subject we need to create a new vector of numbers that we can add to our previous cell state. The equation we use is:
  269. #
  270. # $$ \tilde{c}^{\langle t \rangle} = \tanh(W_c[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_c)\tag{3} $$
  271. #
  272. # Finally, the new cell state is:
  273. #
  274. # $$ c^{\langle t \rangle} = \Gamma_f^{\langle t \rangle}* c^{\langle t-1 \rangle} + \Gamma_u^{\langle t \rangle} *\tilde{c}^{\langle t \rangle} \tag{4} $$
  275. #
  276. #
  277. # #### - Output gate
  278. #
  279. # To decide which outputs we will use, we will use the following two formulas:
  280. #
  281. # $$ \Gamma_o^{\langle t \rangle}=  \sigma(W_o[a^{\langle t-1 \rangle}, x^{\langle t \rangle}] + b_o)\tag{5}$$
  282. # $$ a^{\langle t \rangle} = \Gamma_o^{\langle t \rangle}* \tanh(c^{\langle t \rangle})\tag{6} $$
  283. #
  284. # Where in equation 5 you decide what to output using a sigmoid function and in equation 6 you multiply that by the $\tanh$ of the previous state.
  285.  
  286. # ### 2.1 - LSTM cell
  287. #
  288. # **Exercise**: Implement the LSTM cell described in the Figure (3).
  289. #
  290. # **Instructions**:
  291. # 1. Concatenate $a^{\langle t-1 \rangle}$ and $x^{\langle t \rangle}$ in a single matrix: $concat = \begin{bmatrix} a^{\langle t-1 \rangle} \\ x^{\langle t \rangle} \end{bmatrix}$
  292. # 2. Compute all the formulas 1-6. You can use `sigmoid()` (provided) and `np.tanh()`.
  293. # 3. Compute the prediction $y^{\langle t \rangle}$. You can use `softmax()` (provided).
  294.  
  295. # In[6]:
  296.  
  297. # GRADED FUNCTION: lstm_cell_forward
  298.  
  299. def lstm_cell_forward(xt, a_prev, c_prev, parameters):
  300.     """
  301.    Implement a single forward step of the LSTM-cell as described in Figure (4)
  302.  
  303.    Arguments:
  304.    xt -- your input data at timestep "t", numpy array of shape (n_x, m).
  305.    a_prev -- Hidden state at timestep "t-1", numpy array of shape (n_a, m)
  306.    c_prev -- Memory state at timestep "t-1", numpy array of shape (n_a, m)
  307.    parameters -- python dictionary containing:
  308.                        Wf -- Weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
  309.                        bf -- Bias of the forget gate, numpy array of shape (n_a, 1)
  310.                        Wi -- Weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
  311.                        bi -- Bias of the update gate, numpy array of shape (n_a, 1)
  312.                        Wc -- Weight matrix of the first "tanh", numpy array of shape (n_a, n_a + n_x)
  313.                        bc --  Bias of the first "tanh", numpy array of shape (n_a, 1)
  314.                        Wo -- Weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)
  315.                        bo --  Bias of the output gate, numpy array of shape (n_a, 1)
  316.                        Wy -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
  317.                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
  318.                        
  319.    Returns:
  320.    a_next -- next hidden state, of shape (n_a, m)
  321.    c_next -- next memory state, of shape (n_a, m)
  322.    yt_pred -- prediction at timestep "t", numpy array of shape (n_y, m)
  323.    cache -- tuple of values needed for the backward pass, contains (a_next, c_next, a_prev, c_prev, xt, parameters)
  324.    
  325.    Note: ft/it/ot stand for the forget/update/output gates, cct stands for the candidate value (c tilde),
  326.          c stands for the memory value
  327.    """
  328.  
  329.     # Retrieve parameters from "parameters"
  330.     Wf = parameters["Wf"]
  331.     bf = parameters["bf"]
  332.     Wi = parameters["Wi"]
  333.     bi = parameters["bi"]
  334.     Wc = parameters["Wc"]
  335.     bc = parameters["bc"]
  336.     Wo = parameters["Wo"]
  337.     bo = parameters["bo"]
  338.     Wy = parameters["Wy"]
  339.     by = parameters["by"]
  340.    
  341.     # Retrieve dimensions from shapes of xt and Wy
  342.     n_x, m = xt.shape
  343.     n_y, n_a = Wy.shape
  344.  
  345.     ### START CODE HERE ###
  346.     # Concatenate a_prev and xt (≈3 lines)
  347.     concat = np.zeros((n_x+n_a,m))
  348.     concat[: n_a, :] = a_prev
  349.     concat[n_a :, :] = xt
  350.  
  351.     # Compute values for ft, it, cct, c_next, ot, a_next using the formulas given figure (4) (≈6 lines)
  352.     ft = sigmoid(np.dot(Wf,concat)+bf)
  353.     it = sigmoid(np.dot(Wi,concat)+bi)
  354.     cct = np.tanh(np.dot(Wc,concat)+bc)
  355.     c_next = ft*c_prev+it*cct
  356.     ot = sigmoid(np.dot(Wo,concat)+bo)
  357.     a_next = ot*np.tanh(c_next)
  358.    
  359.     # Compute prediction of the LSTM cell (≈1 line)
  360.     yt_pred = softmax(np.dot(Wy,a_next)+by)
  361.     ### END CODE HERE ###
  362.  
  363.     # store values needed for backward propagation in cache
  364.     cache = (a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters)
  365.  
  366.     return a_next, c_next, yt_pred, cache
  367.  
  368.  
  369. # In[7]:
  370.  
  371. np.random.seed(1)
  372. xt = np.random.randn(3,10)
  373. a_prev = np.random.randn(5,10)
  374. c_prev = np.random.randn(5,10)
  375. Wf = np.random.randn(5, 5+3)
  376. bf = np.random.randn(5,1)
  377. Wi = np.random.randn(5, 5+3)
  378. bi = np.random.randn(5,1)
  379. Wo = np.random.randn(5, 5+3)
  380. bo = np.random.randn(5,1)
  381. Wc = np.random.randn(5, 5+3)
  382. bc = np.random.randn(5,1)
  383. Wy = np.random.randn(2,5)
  384. by = np.random.randn(2,1)
  385.  
  386. parameters = {"Wf": Wf, "Wi": Wi, "Wo": Wo, "Wc": Wc, "Wy": Wy, "bf": bf, "bi": bi, "bo": bo, "bc": bc, "by": by}
  387.  
  388. a_next, c_next, yt, cache = lstm_cell_forward(xt, a_prev, c_prev, parameters)
  389. print("a_next[4] = ", a_next[4])
  390. print("a_next.shape = ", c_next.shape)
  391. print("c_next[2] = ", c_next[2])
  392. print("c_next.shape = ", c_next.shape)
  393. print("yt[1] =", yt[1])
  394. print("yt.shape = ", yt.shape)
  395. print("cache[1][3] =", cache[1][3])
  396. print("len(cache) = ", len(cache))
  397.  
  398.  
  399.  
  400. # ### 2.2 - Forward pass for LSTM
  401. #
  402. # Now that you have implemented one step of an LSTM, you can now iterate this over this using a for-loop to process a sequence of $T_x$ inputs.
  403. #
  404. # <img src="images/LSTM_rnn.png" style="width:500;height:300px;">
  405. # <caption><center> **Figure 4**: LSTM over multiple time-steps. </center></caption>
  406. #
  407. # **Exercise:** Implement `lstm_forward()` to run an LSTM over $T_x$ time-steps.
  408. #
  409. # **Note**: $c^{\langle 0 \rangle}$ is initialized with zeros.
  410.  
  411. # In[8]:
  412.  
  413. # GRADED FUNCTION: lstm_forward
  414.  
  415. def lstm_forward(x, a0, parameters):
  416.     """
  417.    Implement the forward propagation of the recurrent neural network using an LSTM-cell described in Figure (3).
  418.  
  419.    Arguments:
  420.    x -- Input data for every time-step, of shape (n_x, m, T_x).
  421.    a0 -- Initial hidden state, of shape (n_a, m)
  422.    parameters -- python dictionary containing:
  423.                        Wf -- Weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
  424.                        bf -- Bias of the forget gate, numpy array of shape (n_a, 1)
  425.                        Wi -- Weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
  426.                        bi -- Bias of the update gate, numpy array of shape (n_a, 1)
  427.                        Wc -- Weight matrix of the first "tanh", numpy array of shape (n_a, n_a + n_x)
  428.                        bc -- Bias of the first "tanh", numpy array of shape (n_a, 1)
  429.                        Wo -- Weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)
  430.                        bo -- Bias of the output gate, numpy array of shape (n_a, 1)
  431.                        Wy -- Weight matrix relating the hidden-state to the output, numpy array of shape (n_y, n_a)
  432.                        by -- Bias relating the hidden-state to the output, numpy array of shape (n_y, 1)
  433.                        
  434.    Returns:
  435.    a -- Hidden states for every time-step, numpy array of shape (n_a, m, T_x)
  436.    y -- Predictions for every time-step, numpy array of shape (n_y, m, T_x)
  437.    caches -- tuple of values needed for the backward pass, contains (list of all the caches, x)
  438.    """
  439.  
  440.     # Initialize "caches", which will track the list of all the caches
  441.     caches = []
  442.    
  443.     ### START CODE HERE ###
  444.     # Retrieve dimensions from shapes of x and parameters['Wy'] (≈2 lines)
  445.     n_x, m, T_x = x.shape
  446.     n_y, n_a = parameters["Wy"].shape
  447.    
  448.     # initialize "a", "c" and "y" with zeros (≈3 lines)
  449.     a = np.zeros((n_a,m,T_x))
  450.     c = np.zeros((n_a,m,T_x))
  451.     y = np.zeros((n_y,m,T_x))
  452.    
  453.     # Initialize a_next and c_next (≈2 lines)
  454.     a_next = a0
  455.     c_next = np.zeros((n_a,m))
  456.    
  457.     # loop over all time-steps
  458.     for t in range(T_x):
  459.         # Update next hidden state, next memory state, compute the prediction, get the cache (≈1 line)
  460.         a_next, c_next, yt, cache = lstm_cell_forward(x[:,:,t], a_next, c_next, parameters)
  461.         # Save the value of the new "next" hidden state in a (≈1 line)
  462.         a[:,:,t] = a_next
  463.         # Save the value of the prediction in y (≈1 line)
  464.         y[:,:,t] = yt
  465.         # Save the value of the next cell state (≈1 line)
  466.         c[:,:,t]  = c_next
  467.         # Append the cache into caches (≈1 line)
  468.         caches.append(cache)
  469.        
  470.     ### END CODE HERE ###
  471.    
  472.     # store values needed for backward propagation in cache
  473.     caches = (caches, x)
  474.  
  475.     return a, y, c, caches
  476.  
  477.  
  478. # In[9]:
  479.  
  480. np.random.seed(1)
  481. x = np.random.randn(3,10,7)
  482. a0 = np.random.randn(5,10)
  483. Wf = np.random.randn(5, 5+3)
  484. bf = np.random.randn(5,1)
  485. Wi = np.random.randn(5, 5+3)
  486. bi = np.random.randn(5,1)
  487. Wo = np.random.randn(5, 5+3)
  488. bo = np.random.randn(5,1)
  489. Wc = np.random.randn(5, 5+3)
  490. bc = np.random.randn(5,1)
  491. Wy = np.random.randn(2,5)
  492. by = np.random.randn(2,1)
  493.  
  494. parameters = {"Wf": Wf, "Wi": Wi, "Wo": Wo, "Wc": Wc, "Wy": Wy, "bf": bf, "bi": bi, "bo": bo, "bc": bc, "by": by}
  495.  
  496. a, y, c, caches = lstm_forward(x, a0, parameters)
  497. print("a[4][3][6] = ", a[4][3][6])
  498. print("a.shape = ", a.shape)
  499. print("y[1][4][3] =", y[1][4][3])
  500. print("y.shape = ", y.shape)
  501. print("caches[1][1[1]] =", caches[1][1][1])
  502. print("c[1][2][1]", c[1][2][1])
  503. print("len(caches) = ", len(caches))
  504.  
  505.  
  506.  
  507. # Congratulations! You have now implemented the forward passes for the basic RNN and the LSTM. When using a deep learning framework, implementing the forward pass is sufficient to build systems that achieve great performance.
  508. #
  509. # The rest of this notebook is optional, and will not be graded.
  510.  
  511. # ## 3 - Backpropagation in recurrent neural networks (OPTIONAL / UNGRADED)
  512. #
  513. # In modern deep learning frameworks, you only have to implement the forward pass, and the framework takes care of the backward pass, so most deep learning engineers do not need to bother with the details of the backward pass. If however you are an expert in calculus and want to see the details of backprop in RNNs, you can work through this optional portion of the notebook.
  514. #
  515. # When in an earlier course you implemented a simple (fully connected) neural network, you used backpropagation to compute the derivatives with respect to the cost to update the parameters. Similarly, in recurrent neural networks you can to calculate the derivatives with respect to the cost in order to update the parameters. The backprop equations are quite complicated and we did not derive them in lecture. However, we will briefly present them below.
  516.  
  517. # ### 3.1 - Basic RNN  backward pass
  518. #
  519. # We will start by computing the backward pass for the basic RNN-cell.
  520. #
  521. # <img src="images/rnn_cell_backprop.png" style="width:500;height:300px;"> <br>
  522. # <caption><center> **Figure 5**: RNN-cell's backward pass. Just like in a fully-connected neural network, the derivative of the cost function $J$ backpropagates through the RNN by following the chain-rule from calculas. The chain-rule is also used to calculate $(\frac{\partial J}{\partial W_{ax}},\frac{\partial J}{\partial W_{aa}},\frac{\partial J}{\partial b})$ to update the parameters $(W_{ax}, W_{aa}, b_a)$. </center></caption>
  523.  
  524. # #### Deriving the one step backward functions:
  525. #
  526. # To compute the `rnn_cell_backward` you need to compute the following equations. It is a good exercise to derive them by hand.
  527. #
  528. # The derivative of $\tanh$ is $1-\tanh(x)^2$. You can find the complete proof [here](https://www.wyzant.com/resources/lessons/math/calculus/derivative_proofs/tanx). Note that: $ \text{sech}(x)^2 = 1 - \tanh(x)^2$
  529. #
  530. # Similarly for $\frac{ \partial a^{\langle t \rangle} } {\partial W_{ax}}, \frac{ \partial a^{\langle t \rangle} } {\partial W_{aa}},  \frac{ \partial a^{\langle t \rangle} } {\partial b}$, the derivative of  $\tanh(u)$ is $(1-\tanh(u)^2)du$.
  531. #
  532. # The final two equations also follow same rule and are derived using the $\tanh$ derivative. Note that the arrangement is done in a way to get the same dimensions to match.
  533.  
  534. # In[10]:
  535.  
  536. def rnn_cell_backward(da_next, cache):
  537.     """
  538.    Implements the backward pass for the RNN-cell (single time-step).
  539.  
  540.    Arguments:
  541.    da_next -- Gradient of loss with respect to next hidden state
  542.    cache -- python dictionary containing useful values (output of rnn_cell_forward())
  543.  
  544.    Returns:
  545.    gradients -- python dictionary containing:
  546.                        dx -- Gradients of input data, of shape (n_x, m)
  547.                        da_prev -- Gradients of previous hidden state, of shape (n_a, m)
  548.                        dWax -- Gradients of input-to-hidden weights, of shape (n_a, n_x)
  549.                        dWaa -- Gradients of hidden-to-hidden weights, of shape (n_a, n_a)
  550.                        dba -- Gradients of bias vector, of shape (n_a, 1)
  551.    """
  552.    
  553.     # Retrieve values from cache
  554.     (a_next, a_prev, xt, parameters) = cache
  555.    
  556.     # Retrieve values from parameters
  557.     Wax = parameters["Wax"]
  558.     Waa = parameters["Waa"]
  559.     Wya = parameters["Wya"]
  560.     ba = parameters["ba"]
  561.     by = parameters["by"]
  562.  
  563.     ### START CODE HERE ###
  564.     # compute the gradient of tanh with respect to a_next (≈1 line)
  565.     dtanh = (1-a_next*a_next)*da_next
  566.  
  567.     # compute the gradient of the loss with respect to Wax (≈2 lines)
  568.     dxt = np.dot(Wax.T,dtanh)
  569.     dWax = np.dot(dtanh,xt.T)
  570.  
  571.     # compute the gradient with respect to Waa (≈2 lines)
  572.     da_prev = np.dot(Waa.T,dtanh)
  573.     dWaa = np.dot(dtanh,a_prev.T)
  574.  
  575.     # compute the gradient with respect to b (≈1 line)
  576.     dba = np.sum(dtanh,keepdims=True,axis=-1)
  577.  
  578.     ### END CODE HERE ###
  579.    
  580.     # Store the gradients in a python dictionary
  581.     gradients = {"dxt": dxt, "da_prev": da_prev, "dWax": dWax, "dWaa": dWaa, "dba": dba}
  582.    
  583.     return gradients
  584.  
  585.  
  586. # In[11]:
  587.  
  588. np.random.seed(1)
  589. xt = np.random.randn(3,10)
  590. a_prev = np.random.randn(5,10)
  591. Wax = np.random.randn(5,3)
  592. Waa = np.random.randn(5,5)
  593. Wya = np.random.randn(2,5)
  594. b = np.random.randn(5,1)
  595. by = np.random.randn(2,1)
  596. parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "ba": ba, "by": by}
  597.  
  598. a_next, yt, cache = rnn_cell_forward(xt, a_prev, parameters)
  599.  
  600. da_next = np.random.randn(5,10)
  601. gradients = rnn_cell_backward(da_next, cache)
  602. print("gradients[\"dxt\"][1][2] =", gradients["dxt"][1][2])
  603. print("gradients[\"dxt\"].shape =", gradients["dxt"].shape)
  604. print("gradients[\"da_prev\"][2][3] =", gradients["da_prev"][2][3])
  605. print("gradients[\"da_prev\"].shape =", gradients["da_prev"].shape)
  606. print("gradients[\"dWax\"][3][1] =", gradients["dWax"][3][1])
  607. print("gradients[\"dWax\"].shape =", gradients["dWax"].shape)
  608. print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
  609. print("gradients[\"dWaa\"].shape =", gradients["dWaa"].shape)
  610. print("gradients[\"dba\"][4] =", gradients["dba"][4])
  611. print("gradients[\"dba\"].shape =", gradients["dba"].shape)
  612.  
  613.  
  614.  
  615. # #### Backward pass through the RNN
  616. #
  617. # Computing the gradients of the cost with respect to $a^{\langle t \rangle}$ at every time-step $t$ is useful because it is what helps the gradient backpropagate to the previous RNN-cell. To do so, you need to iterate through all the time steps starting at the end, and at each step, you increment the overall $db_a$, $dW_{aa}$, $dW_{ax}$ and you store $dx$.
  618. #
  619. # **Instructions**:
  620. #
  621. # Implement the `rnn_backward` function. Initialize the return variables with zeros first and then loop through all the time steps while calling the `rnn_cell_backward` at each time timestep, update the other variables accordingly.
  622.  
  623. # In[22]:
  624.  
  625. def rnn_backward(da, caches):
  626.     """
  627.    Implement the backward pass for a RNN over an entire sequence of input data.
  628.  
  629.    Arguments:
  630.    da -- Upstream gradients of all hidden states, of shape (n_a, m, T_x)
  631.    caches -- tuple containing information from the forward pass (rnn_forward)
  632.    
  633.    Returns:
  634.    gradients -- python dictionary containing:
  635.                        dx -- Gradient w.r.t. the input data, numpy-array of shape (n_x, m, T_x)
  636.                        da0 -- Gradient w.r.t the initial hidden state, numpy-array of shape (n_a, m)
  637.                        dWax -- Gradient w.r.t the input's weight matrix, numpy-array of shape (n_a, n_x)
  638.                        dWaa -- Gradient w.r.t the hidden state's weight matrix, numpy-arrayof shape (n_a, n_a)
  639.                        dba -- Gradient w.r.t the bias, of shape (n_a, 1)
  640.    """
  641.        
  642.     ### START CODE HERE ###
  643.    
  644.     # Retrieve values from the first cache (t=1) of caches (≈2 lines)
  645.     print("caches shape",np.shape(caches))
  646.     #print("caches",caches)
  647.     #(cache, x) = caches
  648.     (a1, a0, x1, parameters) = caches[0]
  649.    
  650.     # Retrieve dimensions from da's and x1's shapes (≈2 lines)
  651.     n_a, m, T_x = da.shape
  652.     n_x, m = x1.shape
  653.    
  654.     # initialize the gradients with the right sizes (≈6 lines)
  655.     dx = np.zeros((n_x,m,T-x))
  656.     dWax = np.zeros((n_a,n_x))
  657.     dWaa = np.zeros((n_a,n_a))
  658.     dba = np.zeros((n_a,1))
  659.     da0 = np.zeros((n_a,m))
  660.     da_prevt = np.zeros((n_a,m))
  661.    
  662.     # Loop through all the time steps
  663.     for t in reversed(range(None)):
  664.         # Compute gradients at time step t. Choose wisely the "da_next" and the "cache" to use in the backward propagation step. (≈1 line)
  665.         gradients = None
  666.         # Retrieve derivatives from gradients (≈ 1 line)
  667.         dxt, da_prevt, dWaxt, dWaat, dbat = gradients["dxt"], gradients["da_prev"], gradients["dWax"], gradients["dWaa"], gradients["dba"]
  668.         # Increment global derivatives w.r.t parameters by adding their derivative at time-step t (≈4 lines)
  669.         dx[:, :, t] = None
  670.         dWax += None
  671.         dWaa += None
  672.         dba += None
  673.        
  674.     # Set da0 to the gradient of a which has been backpropagated through all time-steps (≈1 line)
  675.     da0 = None
  676.     ### END CODE HERE ###
  677.  
  678.     # Store the gradients in a python dictionary
  679.     gradients = {"dx": dx, "da0": da0, "dWax": dWax, "dWaa": dWaa,"dba": dba}
  680.    
  681.     return gradients
  682.  
  683.  
  684. # In[23]:
  685.  
  686. np.random.seed(1)
  687. x = np.random.randn(3,10,4)
  688. a0 = np.random.randn(5,10)
  689. Wax = np.random.randn(5,3)
  690. Waa = np.random.randn(5,5)
  691. Wya = np.random.randn(2,5)
  692. ba = np.random.randn(5,1)
  693. by = np.random.randn(2,1)
  694. parameters = {"Wax": Wax, "Waa": Waa, "Wya": Wya, "ba": ba, "by": by}
  695. a, y, caches = rnn_forward(x, a0, parameters)
  696. da = np.random.randn(5, 10, 4)
  697. gradients = rnn_backward(da, caches)
  698.  
  699. print("gradients[\"dx\"][1][2] =", gradients["dx"][1][2])
  700. print("gradients[\"dx\"].shape =", gradients["dx"].shape)
  701. print("gradients[\"da0\"][2][3] =", gradients["da0"][2][3])
  702. print("gradients[\"da0\"].shape =", gradients["da0"].shape)
  703. print("gradients[\"dWax\"][3][1] =", gradients["dWax"][3][1])
  704. print("gradients[\"dWax\"].shape =", gradients["dWax"].shape)
  705. print("gradients[\"dWaa\"][1][2] =", gradients["dWaa"][1][2])
  706. print("gradients[\"dWaa\"].shape =", gradients["dWaa"].shape)
  707. print("gradients[\"dba\"][4] =", gradients["dba"][4])
  708. print("gradients[\"dba\"].shape =", gradients["dba"].shape)
  709.  
  710.  
  711. # ## 3.2 - LSTM backward pass
  712.  
  713. # ### 3.2.1 One Step backward
  714. #
  715. # The LSTM backward pass is slighltly more complicated than the forward one. We have provided you with all the equations for the LSTM backward pass below. (If you enjoy calculus exercises feel free to try deriving these from scratch yourself.)
  716. #
  717. # ### 3.2.2 gate derivatives
  718. #
  719. # $$d \Gamma_o^{\langle t \rangle} = da_{next}*\tanh(c_{next}) * \Gamma_o^{\langle t \rangle}*(1-\Gamma_o^{\langle t \rangle})\tag{7}$$
  720. #
  721. # $$d\tilde c^{\langle t \rangle} = dc_{next}*\Gamma_u^{\langle t \rangle}+ \Gamma_o^{\langle t \rangle} (1-\tanh(c_{next})^2) * i_t * da_{next} * \tilde c^{\langle t \rangle} * (1-\tanh(\tilde c)^2) \tag{8}$$
  722. #
  723. # $$d\Gamma_u^{\langle t \rangle} = dc_{next}*\tilde c^{\langle t \rangle} + \Gamma_o^{\langle t \rangle} (1-\tanh(c_{next})^2) * \tilde c^{\langle t \rangle} * da_{next}*\Gamma_u^{\langle t \rangle}*(1-\Gamma_u^{\langle t \rangle})\tag{9}$$
  724. #
  725. # $$d\Gamma_f^{\langle t \rangle} = dc_{next}*\tilde c_{prev} + \Gamma_o^{\langle t \rangle} (1-\tanh(c_{next})^2) * c_{prev} * da_{next}*\Gamma_f^{\langle t \rangle}*(1-\Gamma_f^{\langle t \rangle})\tag{10}$$
  726. #
  727. # ### 3.2.3 parameter derivatives
  728. #
  729. # $$ dW_f = d\Gamma_f^{\langle t \rangle} * \begin{pmatrix} a_{prev} \\ x_t\end{pmatrix}^T \tag{11} $$
  730. # $$ dW_u = d\Gamma_u^{\langle t \rangle} * \begin{pmatrix} a_{prev} \\ x_t\end{pmatrix}^T \tag{12} $$
  731. # $$ dW_c = d\tilde c^{\langle t \rangle} * \begin{pmatrix} a_{prev} \\ x_t\end{pmatrix}^T \tag{13} $$
  732. # $$ dW_o = d\Gamma_o^{\langle t \rangle} * \begin{pmatrix} a_{prev} \\ x_t\end{pmatrix}^T \tag{14}$$
  733. #
  734. # To calculate $db_f, db_u, db_c, db_o$ you just need to sum across the horizontal (axis= 1) axis on $d\Gamma_f^{\langle t \rangle}, d\Gamma_u^{\langle t \rangle}, d\tilde c^{\langle t \rangle}, d\Gamma_o^{\langle t \rangle}$ respectively. Note that you should have the `keep_dims = True` option.
  735. #
  736. # Finally, you will compute the derivative with respect to the previous hidden state, previous memory state, and input.
  737. #
  738. # $$ da_{prev} = W_f^T*d\Gamma_f^{\langle t \rangle} + W_u^T * d\Gamma_u^{\langle t \rangle}+ W_c^T * d\tilde c^{\langle t \rangle} + W_o^T * d\Gamma_o^{\langle t \rangle} \tag{15}$$
  739. # Here, the weights for equations 13 are the first n_a, (i.e. $W_f = W_f[:n_a,:]$ etc...)
  740. #
  741. # $$ dc_{prev} = dc_{next}\Gamma_f^{\langle t \rangle} + \Gamma_o^{\langle t \rangle} * (1- \tanh(c_{next})^2)*\Gamma_f^{\langle t \rangle}*da_{next} \tag{16}$$
  742. # $$ dx^{\langle t \rangle} = W_f^T*d\Gamma_f^{\langle t \rangle} + W_u^T * d\Gamma_u^{\langle t \rangle}+ W_c^T * d\tilde c_t + W_o^T * d\Gamma_o^{\langle t \rangle}\tag{17} $$
  743. # where the weights for equation 15 are from n_a to the end, (i.e. $W_f = W_f[n_a:,:]$ etc...)
  744. #
  745. # **Exercise:** Implement `lstm_cell_backward` by implementing equations $7-17$ below. Good luck! :)
  746.  
  747. # In[ ]:
  748.  
  749. def lstm_cell_backward(da_next, dc_next, cache):
  750.     """
  751.    Implement the backward pass for the LSTM-cell (single time-step).
  752.  
  753.    Arguments:
  754.    da_next -- Gradients of next hidden state, of shape (n_a, m)
  755.    dc_next -- Gradients of next cell state, of shape (n_a, m)
  756.    cache -- cache storing information from the forward pass
  757.  
  758.    Returns:
  759.    gradients -- python dictionary containing:
  760.                        dxt -- Gradient of input data at time-step t, of shape (n_x, m)
  761.                        da_prev -- Gradient w.r.t. the previous hidden state, numpy array of shape (n_a, m)
  762.                        dc_prev -- Gradient w.r.t. the previous memory state, of shape (n_a, m, T_x)
  763.                        dWf -- Gradient w.r.t. the weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
  764.                        dWi -- Gradient w.r.t. the weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
  765.                        dWc -- Gradient w.r.t. the weight matrix of the memory gate, numpy array of shape (n_a, n_a + n_x)
  766.                        dWo -- Gradient w.r.t. the weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)
  767.                        dbf -- Gradient w.r.t. biases of the forget gate, of shape (n_a, 1)
  768.                        dbi -- Gradient w.r.t. biases of the update gate, of shape (n_a, 1)
  769.                        dbc -- Gradient w.r.t. biases of the memory gate, of shape (n_a, 1)
  770.                        dbo -- Gradient w.r.t. biases of the output gate, of shape (n_a, 1)
  771.    """
  772.  
  773.     # Retrieve information from "cache"
  774.     (a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters) = cache
  775.    
  776.     ### START CODE HERE ###
  777.     # Retrieve dimensions from xt's and a_next's shape (≈2 lines)
  778.     n_x, m = None
  779.     n_a, m = None
  780.    
  781.     # Compute gates related derivatives, you can find their values can be found by looking carefully at equations (7) to (10) (≈4 lines)
  782.     dot = None
  783.     dcct = None
  784.     dit = None
  785.     dft = None
  786.    
  787.     # Code equations (7) to (10) (≈4 lines)
  788.     dit = None
  789.     dft = None
  790.     dot = None
  791.     dcct = None
  792.  
  793.     # Compute parameters related derivatives. Use equations (11)-(14) (≈8 lines)
  794.     dWf = None
  795.     dWi = None
  796.     dWc = None
  797.     dWo = None
  798.     dbf = None
  799.     dbi = None
  800.     dbc = None
  801.     dbo = None
  802.  
  803.     # Compute derivatives w.r.t previous hidden state, previous memory state and input. Use equations (15)-(17). (≈3 lines)
  804.     da_prev = None
  805.     dc_prev = None
  806.     dxt = None
  807.     ### END CODE HERE ###
  808.    
  809.     # Save gradients in dictionary
  810.     gradients = {"dxt": dxt, "da_prev": da_prev, "dc_prev": dc_prev, "dWf": dWf,"dbf": dbf, "dWi": dWi,"dbi": dbi,
  811.                 "dWc": dWc,"dbc": dbc, "dWo": dWo,"dbo": dbo}
  812.  
  813.     return gradients
  814.  
  815.  
  816. # In[ ]:
  817.  
  818. np.random.seed(1)
  819. xt = np.random.randn(3,10)
  820. a_prev = np.random.randn(5,10)
  821. c_prev = np.random.randn(5,10)
  822. Wf = np.random.randn(5, 5+3)
  823. bf = np.random.randn(5,1)
  824. Wi = np.random.randn(5, 5+3)
  825. bi = np.random.randn(5,1)
  826. Wo = np.random.randn(5, 5+3)
  827. bo = np.random.randn(5,1)
  828. Wc = np.random.randn(5, 5+3)
  829. bc = np.random.randn(5,1)
  830. Wy = np.random.randn(2,5)
  831. by = np.random.randn(2,1)
  832.  
  833. parameters = {"Wf": Wf, "Wi": Wi, "Wo": Wo, "Wc": Wc, "Wy": Wy, "bf": bf, "bi": bi, "bo": bo, "bc": bc, "by": by}
  834.  
  835. a_next, c_next, yt, cache = lstm_cell_forward(xt, a_prev, c_prev, parameters)
  836.  
  837. da_next = np.random.randn(5,10)
  838. dc_next = np.random.randn(5,10)
  839. gradients = lstm_cell_backward(da_next, dc_next, cache)
  840. print("gradients[\"dxt\"][1][2] =", gradients["dxt"][1][2])
  841. print("gradients[\"dxt\"].shape =", gradients["dxt"].shape)
  842. print("gradients[\"da_prev\"][2][3] =", gradients["da_prev"][2][3])
  843. print("gradients[\"da_prev\"].shape =", gradients["da_prev"].shape)
  844. print("gradients[\"dc_prev\"][2][3] =", gradients["dc_prev"][2][3])
  845. print("gradients[\"dc_prev\"].shape =", gradients["dc_prev"].shape)
  846. print("gradients[\"dWf\"][3][1] =", gradients["dWf"][3][1])
  847. print("gradients[\"dWf\"].shape =", gradients["dWf"].shape)
  848. print("gradients[\"dWi\"][1][2] =", gradients["dWi"][1][2])
  849. print("gradients[\"dWi\"].shape =", gradients["dWi"].shape)
  850. print("gradients[\"dWc\"][3][1] =", gradients["dWc"][3][1])
  851. print("gradients[\"dWc\"].shape =", gradients["dWc"].shape)
  852. print("gradients[\"dWo\"][1][2] =", gradients["dWo"][1][2])
  853. print("gradients[\"dWo\"].shape =", gradients["dWo"].shape)
  854. print("gradients[\"dbf\"][4] =", gradients["dbf"][4])
  855. print("gradients[\"dbf\"].shape =", gradients["dbf"].shape)
  856. print("gradients[\"dbi\"][4] =", gradients["dbi"][4])
  857. print("gradients[\"dbi\"].shape =", gradients["dbi"].shape)
  858. print("gradients[\"dbc\"][4] =", gradients["dbc"][4])
  859. print("gradients[\"dbc\"].shape =", gradients["dbc"].shape)
  860. print("gradients[\"dbo\"][4] =", gradients["dbo"][4])
  861. print("gradients[\"dbo\"].shape =", gradients["dbo"].shape)
  862.  
  863.  
  864.  
  865. # ### 3.3 Backward pass through the LSTM RNN
  866. #
  867. # This part is very similar to the `rnn_backward` function you implemented above. You will first create variables of the same dimension as your return variables. You will then iterate over all the time steps starting from the end and call the one step function you implemented for LSTM at each iteration. You will then update the parameters by summing them individually. Finally return a dictionary with the new gradients.
  868. #
  869. # **Instructions**: Implement the `lstm_backward` function. Create a for loop starting from $T_x$ and going backward. For each step call `lstm_cell_backward` and update the your old gradients by adding the new gradients to them. Note that `dxt` is not updated but is stored.
  870.  
  871. # In[ ]:
  872.  
  873. def lstm_backward(da, caches):
  874.    
  875.     """
  876.    Implement the backward pass for the RNN with LSTM-cell (over a whole sequence).
  877.  
  878.    Arguments:
  879.    da -- Gradients w.r.t the hidden states, numpy-array of shape (n_a, m, T_x)
  880.    dc -- Gradients w.r.t the memory states, numpy-array of shape (n_a, m, T_x)
  881.    caches -- cache storing information from the forward pass (lstm_forward)
  882.  
  883.    Returns:
  884.    gradients -- python dictionary containing:
  885.                        dx -- Gradient of inputs, of shape (n_x, m, T_x)
  886.                        da0 -- Gradient w.r.t. the previous hidden state, numpy array of shape (n_a, m)
  887.                        dWf -- Gradient w.r.t. the weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
  888.                        dWi -- Gradient w.r.t. the weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
  889.                        dWc -- Gradient w.r.t. the weight matrix of the memory gate, numpy array of shape (n_a, n_a + n_x)
  890.                        dWo -- Gradient w.r.t. the weight matrix of the save gate, numpy array of shape (n_a, n_a + n_x)
  891.                        dbf -- Gradient w.r.t. biases of the forget gate, of shape (n_a, 1)
  892.                        dbi -- Gradient w.r.t. biases of the update gate, of shape (n_a, 1)
  893.                        dbc -- Gradient w.r.t. biases of the memory gate, of shape (n_a, 1)
  894.                        dbo -- Gradient w.r.t. biases of the save gate, of shape (n_a, 1)
  895.    """
  896.  
  897.     # Retrieve values from the first cache (t=1) of caches.
  898.     (caches, x) = caches
  899.     (a1, c1, a0, c0, f1, i1, cc1, o1, x1, parameters) = caches[0]
  900.    
  901.     ### START CODE HERE ###
  902.     # Retrieve dimensions from da's and x1's shapes (≈2 lines)
  903.     n_a, m, T_x = None
  904.     n_x, m = None
  905.    
  906.     # initialize the gradients with the right sizes (≈12 lines)
  907.     dx = None
  908.     da0 = None
  909.     da_prevt = None
  910.     dc_prevt = None
  911.     dWf = None
  912.     dWi = None
  913.     dWc = None
  914.     dWo = None
  915.     dbf = None
  916.     dbi = None
  917.     dbc = None
  918.     dbo = None
  919.    
  920.     # loop back over the whole sequence
  921.     for t in reversed(range(None)):
  922.         # Compute all gradients using lstm_cell_backward
  923.         gradients = None
  924.         # Store or add the gradient to the parameters' previous step's gradient
  925.         dx[:,:,t] = None
  926.         dWf = None
  927.         dWi = None
  928.         dWc = None
  929.         dWo = None
  930.         dbf = None
  931.         dbi = None
  932.         dbc = None
  933.         dbo = None
  934.     # Set the first activation's gradient to the backpropagated gradient da_prev.
  935.     da0 = None
  936.    
  937.     ### END CODE HERE ###
  938.  
  939.     # Store the gradients in a python dictionary
  940.     gradients = {"dx": dx, "da0": da0, "dWf": dWf,"dbf": dbf, "dWi": dWi,"dbi": dbi,
  941.                 "dWc": dWc,"dbc": dbc, "dWo": dWo,"dbo": dbo}
  942.    
  943.     return gradients
  944.  
  945.  
  946. # In[ ]:
  947.  
  948. np.random.seed(1)
  949. x = np.random.randn(3,10,7)
  950. a0 = np.random.randn(5,10)
  951. Wf = np.random.randn(5, 5+3)
  952. bf = np.random.randn(5,1)
  953. Wi = np.random.randn(5, 5+3)
  954. bi = np.random.randn(5,1)
  955. Wo = np.random.randn(5, 5+3)
  956. bo = np.random.randn(5,1)
  957. Wc = np.random.randn(5, 5+3)
  958. bc = np.random.randn(5,1)
  959.  
  960. parameters = {"Wf": Wf, "Wi": Wi, "Wo": Wo, "Wc": Wc, "Wy": Wy, "bf": bf, "bi": bi, "bo": bo, "bc": bc, "by": by}
  961.  
  962. a, y, c, caches = lstm_forward(x, a0, parameters)
  963.  
  964. da = np.random.randn(5, 10, 4)
  965. gradients = lstm_backward(da, caches)
  966.  
  967. print("gradients[\"dx\"][1][2] =", gradients["dx"][1][2])
  968. print("gradients[\"dx\"].shape =", gradients["dx"].shape)
  969. print("gradients[\"da0\"][2][3] =", gradients["da0"][2][3])
  970. print("gradients[\"da0\"].shape =", gradients["da0"].shape)
  971. print("gradients[\"dWf\"][3][1] =", gradients["dWf"][3][1])
  972. print("gradients[\"dWf\"].shape =", gradients["dWf"].shape)
  973. print("gradients[\"dWi\"][1][2] =", gradients["dWi"][1][2])
  974. print("gradients[\"dWi\"].shape =", gradients["dWi"].shape)
  975. print("gradients[\"dWc\"][3][1] =", gradients["dWc"][3][1])
  976. print("gradients[\"dWc\"].shape =", gradients["dWc"].shape)
  977. print("gradients[\"dWo\"][1][2] =", gradients["dWo"][1][2])
  978. print("gradients[\"dWo\"].shape =", gradients["dWo"].shape)
  979. print("gradients[\"dbf\"][4] =", gradients["dbf"][4])
  980. print("gradients[\"dbf\"].shape =", gradients["dbf"].shape)
  981. print("gradients[\"dbi\"][4] =", gradients["dbi"][4])
  982. print("gradients[\"dbi\"].shape =", gradients["dbi"].shape)
  983. print("gradients[\"dbc\"][4] =", gradients["dbc"][4])
  984. print("gradients[\"dbc\"].shape =", gradients["dbc"].shape)
  985. print("gradients[\"dbo\"][4] =", gradients["dbo"][4])
  986. print("gradients[\"dbo\"].shape =", gradients["dbo"].shape)
  987.  
  988.  
  989.  
  990. # ### Congratulations !
  991. #
  992. # Congratulations on completing this assignment. You now understand how recurrent neural networks work!
  993. #
  994. # Lets go on to the next exercise, where you'll use an RNN to build a character-level language model.
  995. #
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!