API tools faq
paste
Advertisement
Guest User

Untitled

a guest
Jul 24th, 2016
63
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 12.78 KB | None | 0 0
raw download clone embed print report
  1. I took pytest from `features` branch because it has `--setup-plan` flag
  2. that makes it display the order of setup and teardown of fixtures. This
  3. information is quite useful for understanding what happens in this
  4. issue.
  5.  
  6. I added the following lines to `Metafunc.parametrize` in
  7. `_pytest/python.py` to observe parametrization (I used python 2 for
  8. consistency with the examples from the issue):
  9.  
  10. diff --git a/_pytest/python.py b/_pytest/python.py
  11. index 37151d9..465954f 100644
  12. --- a/_pytest/python.py
  13. +++ b/_pytest/python.py
  14. @@ -763,6 +763,8 @@ class Metafunc(fixtures.FuncargnamesCompatAttr):
  15. It will also override any fixture-function defined scope, allowing
  16. to set a dynamic scope using test context or configuration.
  17. """
  18. + print 'parametrize {} argnames={} indirect={} scope={}'.format(
  19. + self.function.__name__, argnames, indirect, scope)
  20.  
  21. # individual parametrized argument sets can be wrapped in a series
  22. # of markers in which case we unwrap the values and apply the mark
  23.  
  24. Here's the original test (SRC1):
  25.  
  26. import pytest
  27.  
  28. @pytest.yield_fixture(scope="session",
  29. params=["big mac", "whopper"])
  30. def burger(request):
  31. yield request.param
  32.  
  33. @pytest.yield_fixture(scope="function",
  34. params=["curlyfries", "regularfries"])
  35. def fries(request):
  36. yield request.param
  37.  
  38. def test_burgers(burger):
  39. print "test_burgers: {0}".format(burger)
  40.  
  41. def test_combo1(burger, fries):
  42. print("test_combo1: {0}{1}".format(burger, fries))
  43.  
  44. It produces the following output (OUT1):
  45.  
  46. $ pytest --setup-only -s test_fastfood.py
  47. ...
  48. parametrize test_burgers argnames=burger indirect=True scope=session
  49. parametrize test_combo1 argnames=burger indirect=True scope=session
  50. parametrize test_combo1 argnames=fries indirect=True scope=function
  51. collected 6 items
  52.  
  53. test_fastfood.py
  54. SETUP S burger[big mac]
  55. test_fastfood.py::test_burgers[big mac] (fixtures used: burger)
  56. SETUP F fries[curlyfries]
  57. test_fastfood.py::test_combo1[big mac-curlyfries] (fixtures used: burger, fries)
  58. TEARDOWN F fries[curlyfries]
  59. SETUP F fries[regularfries]
  60. test_fastfood.py::test_combo1[big mac-regularfries] (fixtures used: burger, fries)
  61. TEARDOWN F fries[regularfries]
  62. TEARDOWN S burger[big mac]
  63. SETUP S burger[whopper]
  64. test_fastfood.py::test_burgers[whopper] (fixtures used: burger)
  65. SETUP F fries[curlyfries]
  66. test_fastfood.py::test_combo1[whopper-curlyfries] (fixtures used: burger, fries)
  67. TEARDOWN F fries[curlyfries]
  68. SETUP F fries[regularfries]
  69. test_fastfood.py::test_combo1[whopper-regularfries] (fixtures used: burger, fries)
  70. TEARDOWN F fries[regularfries]
  71. TEARDOWN S burger[whopper]
  72.  
  73. We can see that `parametrize` is called with `scope` set to `"session"`
  74. and `indirect` set to `True`. As expected, the tests are ordered in a
  75. resonable way and the session scoped fixture is created twice: once
  76. for each value of the parameter.
  77.  
  78. Then I removed the parameters from fixture decorator and added the
  79. `pytest_generate_tests` function that parametrizes `burger` fixture
  80. programmatically to get the second example from the issue (SRC2):
  81.  
  82. import pytest
  83.  
  84. def pytest_generate_tests(metafunc):
  85. if "burger" in metafunc.fixturenames:
  86. metafunc.parametrize("burger", argvalues=["big mac", "whopper"])
  87.  
  88. @pytest.yield_fixture(scope="session")
  89. def burger(request):
  90. yield request.param
  91.  
  92. # the rest is the same as before
  93.  
  94. This produces the following output (OUT2):
  95.  
  96. $ pytest --setup-only -s test_fastfood.py
  97. ...
  98. parametrize test_burgers argnames=burger indirect=False scope=None
  99. parametrize test_combo1 argnames=burger indirect=False scope=None
  100. parametrize test_combo1 argnames=fries indirect=True scope=function
  101. collected 6 items
  102.  
  103. test_fastfood.py
  104. SETUP F burger[big mac]
  105. test_fastfood.py::test_burgers[big mac] (fixtures used: burger)
  106. TEARDOWN F burger[big mac]
  107. SETUP F burger[whopper]
  108. test_fastfood.py::test_burgers[whopper] (fixtures used: burger)
  109. TEARDOWN F burger[whopper]
  110. SETUP F burger[big mac]
  111. SETUP F fries[curlyfries]
  112. test_fastfood.py::test_combo1[big mac-curlyfries] (fixtures used: burger, fries)
  113. TEARDOWN F fries[curlyfries]
  114. TEARDOWN F burger[big mac]
  115. SETUP F burger[big mac]
  116. SETUP F fries[regularfries]
  117. test_fastfood.py::test_combo1[big mac-regularfries] (fixtures used: burger, fries)
  118. TEARDOWN F fries[regularfries]
  119. TEARDOWN F burger[big mac]
  120. SETUP F burger[whopper]
  121. SETUP F fries[curlyfries]
  122. test_fastfood.py::test_combo1[whopper-curlyfries] (fixtures used: burger, fries)
  123. TEARDOWN F fries[curlyfries]
  124. TEARDOWN F burger[whopper]
  125. SETUP F burger[whopper]
  126. SETUP F fries[regularfries]
  127. test_fastfood.py::test_combo1[whopper-regularfries] (fixtures used: burger, fries)
  128. TEARDOWN F fries[regularfries]
  129. TEARDOWN F burger[whopper]
  130.  
  131. We can see that `burger` fixture becomes function scope (the `F` letter
  132. after `SETUP` and `TEARDOWN`) and is now created and destroyed for each
  133. test item. The order of the tests changes but it makes sense given that
  134. the fixture is now function scoped.
  135.  
  136. Above we can see that the scope parameter of `metafunc.parametrize` is
  137. `None` for `burger` fixtures. Let's set it to `"session"` (SRC3):
  138.  
  139. def pytest_generate_tests(metafunc):
  140. if "burger" in metafunc.fixturenames:
  141. metafunc.parametrize("burger", argvalues=["big mac", "whopper"],
  142. scope="session")
  143.  
  144. This produces the following output (OUT3):
  145.  
  146. $ pytest --setup-only -s test_fastfood.py
  147. ...
  148. parametrize test_burgers argnames=burger indirect=False scope=session
  149. parametrize test_combo1 argnames=burger indirect=False scope=session
  150. parametrize test_combo1 argnames=fries indirect=True scope=function
  151. collected 6 items
  152.  
  153. test_fastfood.py
  154. SETUP S burger[big mac]
  155. test_fastfood.py::test_burgers[big mac] (fixtures used: burger)
  156. SETUP F fries[curlyfries]
  157. test_fastfood.py::test_combo1[big mac-curlyfries] (fixtures used: burger, fries)
  158. TEARDOWN F fries[curlyfries]
  159. TEARDOWN S burger[big mac]
  160. SETUP S burger[whopper]
  161. test_fastfood.py::test_burgers[whopper] (fixtures used: burger)
  162. SETUP F fries[regularfries]
  163. test_fastfood.py::test_combo1[big mac-regularfries] (fixtures used: burger, fries)
  164. TEARDOWN F fries[regularfries]
  165. TEARDOWN S burger[whopper]
  166. SETUP S burger[whopper]
  167. SETUP F fries[curlyfries]
  168. test_fastfood.py::test_combo1[whopper-curlyfries] (fixtures used: burger, fries)
  169. TEARDOWN F fries[curlyfries]
  170. TEARDOWN S burger[whopper]
  171. SETUP S burger[whopper]
  172. SETUP F fries[regularfries]
  173. test_fastfood.py::test_combo1[whopper-regularfries] (fixtures used: burger, fries)
  174. TEARDOWN F fries[regularfries]
  175. TEARDOWN S burger[whopper]
  176.  
  177. This seems strange, but actually it sort of makes sense. Our
  178. parametrization of the tests is direct, so what happens is that a
  179. separate session scoped virtual fixture called `burger` is created for
  180. each test function. The test suite uses two separate session scoped
  181. fixtures both called `burger` and that confuses the sorting of the tests
  182. as well as the code that does `--setup-plan`. I don't really see the
  183. purpose of this setup, so I wouldn't necessarily call this a bug.
  184.  
  185. Let's make parametrization indirect to bring back some sanity (SRC4):
  186.  
  187. def pytest_generate_tests(metafunc):
  188. if "burger" in metafunc.fixturenames:
  189. metafunc.parametrize("burger", argvalues=["big mac", "whopper"],
  190. indirect=True, scope="session")
  191.  
  192. Run pytest again (OUT4):
  193.  
  194. $ pytest --setup-only -s test_fastfood.py
  195. ...
  196. parametrize test_burgers argnames=burger indirect=True scope=session
  197. parametrize test_combo1 argnames=burger indirect=True scope=session
  198. parametrize test_combo1 argnames=fries indirect=True scope=function
  199. collected 6 items
  200.  
  201. test_fastfood.py
  202. SETUP S burger[big mac]
  203. test_fastfood.py::test_burgers[big mac] (fixtures used: burger)
  204. SETUP F fries[curlyfries]
  205. test_fastfood.py::test_combo1[big mac-curlyfries] (fixtures used: burger, fries)
  206. TEARDOWN F fries[curlyfries]
  207. SETUP F fries[regularfries]
  208. test_fastfood.py::test_combo1[big mac-regularfries] (fixtures used: burger, fries)
  209. TEARDOWN F fries[regularfries]
  210. TEARDOWN S burger[big mac]
  211. SETUP S burger[whopper]
  212. test_fastfood.py::test_burgers[whopper] (fixtures used: burger)
  213. SETUP F fries[curlyfries]
  214. test_fastfood.py::test_combo1[whopper-curlyfries] (fixtures used: burger, fries)
  215. TEARDOWN F fries[curlyfries]
  216. SETUP F fries[regularfries]
  217. test_fastfood.py::test_combo1[whopper-regularfries] (fixtures used: burger, fries)
  218. TEARDOWN F fries[regularfries]
  219. TEARDOWN S burger[whopper]
  220.  
  221. This is the same ouput as OUT1, so sorting now works correctly.
  222.  
  223. Another variation worth noting is parametrization via
  224. `pytest.mark.parametrize` decorator (SRC5):
  225.  
  226. import pytest
  227.  
  228. @pytest.yield_fixture(scope="session")
  229. def burger(request):
  230. yield request.param
  231.  
  232. @pytest.yield_fixture(scope="function",
  233. params=["curlyfries", "regularfries"])
  234. def fries(request):
  235. yield request.param
  236.  
  237. @pytest.mark.parametrize("burger", ["big mac", "whopper"])
  238. def test_burgers(burger):
  239. print "\ntest_burgers: {0}".format(burger)
  240.  
  241. @pytest.mark.parametrize("burger", ["big mac", "whopper"])
  242. def test_combo1(burger, fries):
  243. print("\ntest_combo1: {0}{1}".format(burger, fries))
  244.  
  245. This, quite predictably, produces OUT2. Adding `scope="session"` to
  246. the arguments of `parametrize` brings us to OUT3 and further adding
  247. `indirect=True` brings us to OUT4/1. Everything is consistent here.
  248.  
  249. Just for completeness of the picture let's do another experiment (SRC6):
  250.  
  251. @pytest.mark.parametrize("burger", ["big mac", "whopper"], indirect=True)
  252.  
  253. This mean that we parametrize our session scoped `burger` fixture in the
  254. function scope (whatever that's supposed to mean). Here's the output
  255. (OUT6):
  256.  
  257. parametrize test_burgers argnames=burger indirect=True scope=None
  258. parametrize test_combo1 argnames=fries indirect=True scope=function
  259. parametrize test_combo1 argnames=burger indirect=True scope=None
  260. collected 6 items
  261.  
  262. test_fastfood.py
  263. SETUP S burger[big mac]
  264. test_fastfood.py::test_burgers[big mac] (fixtures used: burger)
  265. TEARDOWN S burger[big mac]
  266. SETUP S burger[whopper]
  267. test_fastfood.py::test_burgers[whopper] (fixtures used: burger)
  268. TEARDOWN S burger[whopper]
  269. SETUP S burger[small mac]
  270. SETUP F fries[curlyfries]
  271. test_fastfood.py::test_combo1[curlyfries-small mac] (fixtures used: burger, fries)
  272. TEARDOWN F fries[curlyfries]
  273. TEARDOWN S burger[small mac]
  274. SETUP S burger[whooper]
  275. SETUP F fries[curlyfries]
  276. test_fastfood.py::test_combo1[curlyfries-whooper] (fixtures used: burger, fries)
  277. TEARDOWN F fries[curlyfries]
  278. TEARDOWN S burger[whooper]
  279. SETUP S burger[small mac]
  280. SETUP F fries[regularfries]
  281. test_fastfood.py::test_combo1[regularfries-small mac] (fixtures used: burger, fries)
  282. TEARDOWN F fries[regularfries]
  283. TEARDOWN S burger[small mac]
  284. SETUP S burger[whooper]
  285. SETUP F fries[regularfries]
  286. test_fastfood.py::test_combo1[regularfries-whooper] (fixtures used: burger, fries)
  287. TEARDOWN F fries[regularfries]
  288. TEARDOWN S burger[whooper]
  289.  
  290. We've effectively turned our session scoped fixture into a function
  291. scoped fixture (it's created and destroyed for each test item) even
  292. though it's corresponding `FixtureDef` still thinks it's session scope.
  293.  
  294. ## Conclusion
  295.  
  296. The ordering works correctly regardless of parametrization method.
  297. Parametrization APIs, however, are confusing:
  298.  
  299. - `metafunc.parametrize` and `pytest.mark.parametrize` allow
  300. parametrization of non-function scoped fixtures at the function level,
  301. - the above behaviour is the default (changing the default would also
  302. be bad for the reason stated in the next item of this list)
  303. - on top of this, if the scope is set correctly (e.g. to `"session"`)
  304. via `scope` parameter but the parameter list is not identical between
  305. different functions, things get very confusing.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!