SHARE
TWEET

Untitled

a guest Oct 7th, 2014 938 Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. Arc doesn't cause the expired link issue. Arc is just a Lisp that lets
  2. you write your program however you want. The HN software is written in
  3. a style that uses closures to remember what a user is up to across
  4. multiple web requests. I'll describe a classic example of this, then
  5. how we changed it and why.
  6.  
  7. When you click "More" at the bottom of a page, HN shows you the next
  8. 30 stories. These are different for logged-in users--they're affected
  9. by profile settings like "showdead"--so, if you're logged in, the
  10. system needs to compute the next 30 stories to show you. Which 30
  11. those are depends on the ones you were just looking at. So the server
  12. needs to know what you were just looking at in order to handle the
  13. "More" request correctly. How does it know that? There are many ways.
  14. HN's traditional way is to make a closure (a function that remembers
  15. any info it needs) at the time that it's generating the original page,
  16. which, if and when it is called, will compute the correct next 30
  17. stories.
  18.  
  19. The advantage of doing it this way is that, at the time you're making
  20. that closure, all the information you need to process the next request
  21. correctly is already right there in scope. You don't have to remember
  22. it, reconstitute it, or anything--all of which takes extra code, and
  23. different code for each kind of request.
  24.  
  25. You can't send the closure directly to the web browser, so you make a
  26. unique ID instead, save that closure in a table keyed by the ID, and
  27. put the ID in a link. When the user clicks that link, the server gets
  28. the ID, looks up the closure, and executes it.
  29.  
  30. If you're saving lots of closures, eventually you'll run out of RAM if
  31. you don't garbage-collect some. So the HN server periodically prunes
  32. the oldest ones. If one of those links is still open in a browser
  33. window somewhere and the user clicks on it, the server will no longer
  34. remember what to do. That is when HN says "expired link".
  35.  
  36. As HN grew larger, there were many more of these closures, and the
  37. odds were higher that someone would try to use one after it was
  38. pruned. A few months ago, since we were working on that part of the
  39. code anyway, we decided to eliminate the most common cases. We
  40. measured all the different kinds of closure in the table and replaced
  41. the most common ones with more traditional ways of passing state back
  42. to the server, like query strings and hidden form fields. We did that
  43. until the total number of closures was down by an order of magnitude,
  44. and then--equally importantly--we stopped. Now the system has enough
  45. RAM to remember the vast majority of closures again, and the "expired
  46. link" errors have gone down to a tiny fraction of what they were
  47. before. (The one unfortunate exception to this is when we restart the
  48. server process.)
  49.  
  50. I was the person who changed the code to not use closures for "More"
  51. links and other common cases, and it gave me a new appreciation for
  52. how powerful the closure technique is. Getting rid of those closures
  53. was a pain, and it made the code more complicated. When HN was first
  54. being built, the tradeoff in favor of programming ease was a
  55. no-brainer. In fact, it's because of methods like this that HN exists
  56. at all, because HN wouldn't exist if one person (pg) hadn't been able
  57. to build it in his spare time. The fact that it lasted as a one-man
  58. side project for something like 7 years is evidence of how powerful
  59. its software design is. The closure technique is one piece of that,
  60. and it's still used throughout the codebase. It would be foolish to
  61. try to replace all of them; the same tradeoff in favor of programming
  62. efficiency is still the dominant factor in most of those cases because
  63. they are relatively little used.
RAW Paste Data
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy. OK, I Understand
Top