API tools faq
paste
Advertisement
Guest User

Facebook Robots Leaked By Fallag Iheb

a guest
Aug 6th, 2014
459
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 3.24 KB | None | 0 0
raw download clone embed print report
  1. Facebook Robots Leaked By Fallag Iheb
  2. Fallaga Team !!
  3. # Notice: Crawling Facebook is prohibited unless you have express written
  4. # permission. See: http://www.facebook.com/apps/site_scraping_tos_terms.php
  5.  
  6. User-agent: baiduspider
  7. Disallow: /ajax/
  8. Disallow: /album.php
  9. Disallow: /checkpoint/
  10. Disallow: /contact_importer/
  11. Disallow: /feeds/
  12. Disallow: /file_download.php
  13. Disallow: /l.php
  14. Disallow: /p.php
  15. Disallow: /photo.php
  16. Disallow: /photo_comments.php
  17. Disallow: /photo_search.php
  18. Disallow: /photos.php
  19. Disallow: /sharer/
  20.  
  21. User-agent: Googlebot
  22. Disallow: /ajax/
  23. Disallow: /album.php
  24. Disallow: /checkpoint/
  25. Disallow: /contact_importer/
  26. Disallow: /feeds/
  27. Disallow: /file_download.php
  28. Disallow: /l.php
  29. Disallow: /p.php
  30. Disallow: /photo.php
  31. Disallow: /photo_comments.php
  32. Disallow: /photo_search.php
  33. Disallow: /photos.php
  34. Disallow: /sharer/
  35.  
  36. User-agent: msnbot
  37. Disallow: /ajax/
  38. Disallow: /album.php
  39. Disallow: /checkpoint/
  40. Disallow: /contact_importer/
  41. Disallow: /feeds/
  42. Disallow: /file_download.php
  43. Disallow: /l.php
  44. Disallow: /p.php
  45. Disallow: /photo.php
  46. Disallow: /photo_comments.php
  47. Disallow: /photo_search.php
  48. Disallow: /photos.php
  49. Disallow: /sharer/
  50.  
  51. User-agent: Naverbot
  52. Disallow: /ajax/
  53. Disallow: /album.php
  54. Disallow: /checkpoint/
  55. Disallow: /contact_importer/
  56. Disallow: /feeds/
  57. Disallow: /file_download.php
  58. Disallow: /l.php
  59. Disallow: /p.php
  60. Disallow: /photo.php
  61. Disallow: /photo_comments.php
  62. Disallow: /photo_search.php
  63. Disallow: /photos.php
  64. Disallow: /sharer/
  65.  
  66. User-agent: seznambot
  67. Disallow: /ajax/
  68. Disallow: /album.php
  69. Disallow: /checkpoint/
  70. Disallow: /contact_importer/
  71. Disallow: /feeds/
  72. Disallow: /file_download.php
  73. Disallow: /l.php
  74. Disallow: /p.php
  75. Disallow: /photo.php
  76. Disallow: /photo_comments.php
  77. Disallow: /photo_search.php
  78. Disallow: /photos.php
  79. Disallow: /sharer/
  80.  
  81. User-agent: Slurp
  82. Disallow: /ajax/
  83. Disallow: /album.php
  84. Disallow: /checkpoint/
  85. Disallow: /contact_importer/
  86. Disallow: /feeds/
  87. Disallow: /file_download.php
  88. Disallow: /l.php
  89. Disallow: /p.php
  90. Disallow: /photo.php
  91. Disallow: /photo_comments.php
  92. Disallow: /photo_search.php
  93. Disallow: /photos.php
  94. Disallow: /sharer/
  95.  
  96. User-agent: teoma
  97. Disallow: /ajax/
  98. Disallow: /album.php
  99. Disallow: /checkpoint/
  100. Disallow: /contact_importer/
  101. Disallow: /feeds/
  102. Disallow: /file_download.php
  103. Disallow: /l.php
  104. Disallow: /p.php
  105. Disallow: /photo.php
  106. Disallow: /photo_comments.php
  107. Disallow: /photo_search.php
  108. Disallow: /photos.php
  109. Disallow: /sharer/
  110.  
  111. User-agent: Yandex
  112. Disallow: /ajax/
  113. Disallow: /album.php
  114. Disallow: /checkpoint/
  115. Disallow: /contact_importer/
  116. Disallow: /feeds/
  117. Disallow: /file_download.php
  118. Disallow: /l.php
  119. Disallow: /p.php
  120. Disallow: /photo.php
  121. Disallow: /photo_comments.php
  122. Disallow: /photo_search.php
  123. Disallow: /photos.php
  124. Disallow: /sharer/
  125.  
  126. User-agent: Yeti
  127. Disallow: /ajax/
  128. Disallow: /album.php
  129. Disallow: /checkpoint/
  130. Disallow: /contact_importer/
  131. Disallow: /feeds/
  132. Disallow: /file_download.php
  133. Disallow: /l.php
  134. Disallow: /p.php
  135. Disallow: /photo.php
  136. Disallow: /photo_comments.php
  137. Disallow: /photo_search.php
  138. Disallow: /photos.php
  139. Disallow: /sharer/
  140.  
  141. User-agent: ia_archiver
  142. Allow: /about/privacy
  143. Allow: /full_data_use_policy
  144. Allow: /legal/terms
  145. Allow: /policy.php
  146. Disallow: /
  147.  
  148. User-agent: *
  149. Disallow: /
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!