API tools faq
paste
Bananaware

shallow-conv adventures

Bananaware
Jul 10th, 2020
87
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 6.86 KB | None | 0 0
raw download clone embed print report
  1. tl;dr:
  2. convoltional > dense shallow, mas sem ganho real de performance nos testes (modelos começam a dar overfit sem ganho em valid/test). precisa de mais imgs? data augmentation (flips/whatever)?
  3. melhor modelo (considerando a ROC) foi CNN_4F5K144_G7 0.772 precision, 0.753 accuracy.
  4.  
  5. 10-fold cross-validation ROC tradeoffs (https://i.imgur.com/y52DA4V.png):
  6. 1% FP -> 60.1% TP
  7. 3% FP -> 69.8% TP
  8. 5% FP -> 74.7% TP
  9. 9% FP -> 80.3% TP
  10.  
  11. provavelmente razoável aceitar 3% FP, se usar esse como modelo final?
  12.  
  13.  
  14. garbage in -> garbage out? como avaliar essas imgs?
  15. https://imgur.com/a/yd1NDDv
  16.  
  17.  
  18.  
  19.  
  20. atual split de imagens boas/ruins:
  21. Jersey_1 | Good: 575 | Bad: 92 | 86.21% (575/667)
  22. Jersey_2 | Good: 203 | Bad: 91 | 69.05% (203/294)
  23. Jersey_3 | Good: 408 | Bad: 90 | 81.93% (408/498)
  24. Puruna_1-P1 | Good: 51 | Bad: 376 | 11.94% (51/427)
  25. Puruna_1-P2 | Good: 58 | Bad: 390 | 12.95% (58/448)
  26. Puruna_2 | Good: 129 | Bad: 1167 | 9.954% (129/1296)
  27. Puruna_3 | Good: 123 | Bad: 1670 | 6.86% (123/1793)
  28. USP-P1 | Good: 307 | Bad: 28 | 91.64% (307/335)
  29. USP-P2 | Good: 265 | Bad: 45 | 85.48% (265/310)
  30. USP-P3 | Good: 325 | Bad: 17 | 95.03% (325/342)
  31. Total | Good: 2444 | Bad: 3966 | 38.13% (2444/6410)
  32.  
  33.  
  34.  
  35. mais detalhes do conv training:
  36.  
  37. modelos simples (modelo final 242 KB!):
  38.  
  39. Model: "conv_simple"
  40. _________________________________________________________________
  41. Layer (type) Output Shape Param #
  42. =================================================================
  43. conv2d_1 (Conv2D) (None, 140, 140, 4) 104
  44. _________________________________________________________________
  45. max_pooling2d_1 (MaxPooling2 (None, 70, 70, 4) 0
  46. _________________________________________________________________
  47. conv2d_2 (Conv2D) (None, 66, 66, 8) 808
  48. _________________________________________________________________
  49. max_pooling2d_2 (MaxPooling2 (None, 33, 33, 8) 0
  50. _________________________________________________________________
  51. flatten_1 (Flatten) (None, 8712) 0
  52. _________________________________________________________________
  53. dense_1 (Dense) (None, 2) 17426
  54. =================================================================
  55. Total params: 18,338
  56. Trainable params: 18,338
  57. Non-trainable params: 0
  58.  
  59.  
  60. alguns plots de treinamento:
  61. https://i.imgur.com/gjnPft7.png
  62. https://i.imgur.com/e4rlvJF.png
  63. https://i.imgur.com/QUlPmc0.png
  64. https://i.imgur.com/PpiSlDB.png
  65. https://i.imgur.com/eLE0xa0.png
  66.  
  67.  
  68. testei vários modelos pequenos e alguns poucos maiores (conv, connected). overfit, ou atinge maxima sem melhorar valid/teste, ou o que seja... melhores modelos shallow-conv.
  69. tentei resolver o overfit diminuindo a capacidade do modelo (input size, tamanho dos filtros), não ajudou de verdade.
  70. dropout também não melhorou, não que eu esperasse que fosse ajudar com um modelo tão shallow... (se bem que agora veio a ideia, faz um razoavelmente deep e coloca dropouts? parece não valer muito a pena)
  71.  
  72.  
  73.  
  74.  
  75. callbacks para pegar os modelos mais razoáveis (pre-superoverfit).
  76. alguns logs dos melhores modelos (melhor modelo no final do arquivo):
  77.  
  78. Currently training: CNN_4F5K144_F9.h5
  79.  
  80. explicação do naming:
  81. 4F = 4 filtros na primeira camada
  82. 5K = kernel 5x5
  83. F = split train/valid/test F (eu tentei várias divisões... chamei de A, B, C, D, E, F, G. definição do F nas linhas a seguir)
  84. 9 = número de epochs
  85.  
  86. SPLIT_F = [[DF+'Puruna_2', DF+'Puruna_3', DF+'Jersey_2', DF+'USP-P1'],
  87. [DF+'Puruna_1-P1', DF+'Jersey_3', DF+'USP-P3'],
  88. [DF+'Puruna_1-P2', DF+'Jersey_1', DF+'USP-P2']]
  89. train 3718, valid 1267, test 1425
  90. test ratio = 898/1425, 63% boas
  91.  
  92.  
  93. Train on 3718 samples, validate on 1267 samples
  94. 3718/3718 [==============================] - 1s 228us/step - loss: 0.2085 - accuracy: 0.9236 - val_loss: 0.4343 - val_accuracy: 0.8114
  95.  
  96. Acc: 0.776 | Pre: 0.854 | Sample: 143
  97. Acc: 0.748 | Pre: 0.821 | Sample: 143
  98. Acc: 0.762 | Pre: 0.850 | Sample: 143
  99. Acc: 0.769 | Pre: 0.890 | Sample: 143
  100. Acc: 0.762 | Pre: 0.826 | Sample: 143
  101. Acc: 0.739 | Pre: 0.827 | Sample: 142
  102. Acc: 0.768 | Pre: 0.852 | Sample: 142
  103. Acc: 0.704 | Pre: 0.718 | Sample: 142
  104. Acc: 0.725 | Pre: 0.719 | Sample: 142
  105. Acc: 0.599 | Pre: 0.629 | Sample: 142
  106. ==
  107. EndAcc: 0.735 | EndPre: 0.799 | Sample: 1425
  108.  
  109. ROC (AUC = 0.88 +- 0.07):
  110. https://i.imgur.com/E1osnBs.png
  111.  
  112.  
  113. Currently training: CNN_4F5K144_F8.h5
  114. Train on 3718 samples, validate on 1267 samples
  115. 3718/3718 [==============================] - 1s 218us/step - loss: 0.1975 - accuracy: 0.9317 - val_loss: 0.5795 - val_accuracy: 0.8106
  116.  
  117. Acc: 0.643 | Pre: 0.767 | Sample: 143
  118. Acc: 0.769 | Pre: 0.835 | Sample: 143
  119. Acc: 0.727 | Pre: 0.823 | Sample: 143
  120. Acc: 0.797 | Pre: 0.896 | Sample: 143
  121. Acc: 0.699 | Pre: 0.770 | Sample: 143
  122. Acc: 0.697 | Pre: 0.813 | Sample: 142
  123. Acc: 0.697 | Pre: 0.831 | Sample: 142
  124. Acc: 0.718 | Pre: 0.731 | Sample: 142
  125. Acc: 0.704 | Pre: 0.708 | Sample: 142
  126. Acc: 0.627 | Pre: 0.645 | Sample: 142
  127. ==
  128. EndAcc: 0.708 | EndPre: 0.782 | Sample: 1425
  129.  
  130. ROC (AUC = 0.87 +- 0.06):
  131. https://i.imgur.com/qrobF3t.png
  132.  
  133.  
  134.  
  135.  
  136. SPLIT_G = [[DF+'Puruna_1-P1', DF+'Puruna_3', DF+'Jersey_2', DF+'USP-P2'],
  137. [DF+'Puruna_1-P2', DF+'Jersey_1', DF+'USP-P3'],
  138. [DF+'Puruna_2', DF+'USP-P1', DF+'Jersey_3']]
  139. train 2824, valid 1457, test 2129
  140. test ratio = 844/2129, 39% boas
  141.  
  142. Currently training: CNN_4F5K144_G3.h5
  143. Train on 2824 samples, validate on 1457 samples
  144. 2824/2824 [==============================] - 1s 219us/step - loss: 0.3277 - accuracy: 0.8792 - val_loss: 0.5593 - val_accuracy: 0.7467
  145.  
  146. Acc: 0.629 | Pre: 0.636 | Sample: 213
  147. Acc: 0.728 | Pre: 0.800 | Sample: 213
  148. Acc: 0.831 | Pre: 0.802 | Sample: 213
  149. Acc: 0.883 | Pre: 0.849 | Sample: 213
  150. Acc: 0.850 | Pre: 0.817 | Sample: 213
  151. Acc: 0.850 | Pre: 0.833 | Sample: 213
  152. Acc: 0.822 | Pre: 0.735 | Sample: 213
  153. Acc: 0.873 | Pre: 0.852 | Sample: 213
  154. Acc: 0.822 | Pre: 0.756 | Sample: 213
  155. Acc: 0.448 | Pre: 0.408 | Sample: 212
  156. ==
  157. EndAcc: 0.773 | EndPre: 0.749 | Sample: 2129
  158.  
  159. ROC (AUC = 0.90 +- 0.15):
  160. https://i.imgur.com/4jIkdzN.png
  161.  
  162.  
  163.  
  164. Currently training: CNN_4F5K144_G7.h5
  165. Train on 2824 samples, validate on 1457 samples
  166. 2824/2824 [==============================] - 1s 219us/step - loss: 0.2121 - accuracy: 0.9249 - val_loss: 0.5539 - val_accuracy: 0.7454
  167.  
  168. Acc: 0.685 | Pre: 0.781 | Sample: 213
  169. Acc: 0.728 | Pre: 0.737 | Sample: 213
  170. Acc: 0.831 | Pre: 0.810 | Sample: 213
  171. Acc: 0.817 | Pre: 0.780 | Sample: 213
  172. Acc: 0.817 | Pre: 0.817 | Sample: 213
  173. Acc: 0.859 | Pre: 0.814 | Sample: 213
  174. Acc: 0.836 | Pre: 0.769 | Sample: 213
  175. Acc: 0.859 | Pre: 0.821 | Sample: 213
  176. Acc: 0.854 | Pre: 0.798 | Sample: 213
  177. Acc: 0.434 | Pre: 0.403 | Sample: 212
  178. ==
  179. EndAcc: 0.772 | EndPre: 0.753 | Sample: 2129
  180.  
  181. ROC (AUC = 0.90 +- 0.14):
  182. https://i.imgur.com/y52DA4V.png
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!