Pastebin
API
tools
faq
paste
Login
Sign up
SHARE
TWEET
Untitled
a guest
May 16th, 2018
131
0
Never
Add comment
Not a member of Pastebin yet?
Sign Up
, it unlocks many cool features!
text
0.33 KB
| None
|
0
0
raw
download
clone
embed
print
report
어떤 웹 페이지에 대한 웹크롤링을 제한시키기 위한 약속같은것으로써 웹사이트의 최상위 경로에 txt파일로 적어서 저장해 둔다.
모든 경로에 봇의 접근을 허락하는 경우
User-agent: *
Allow: /
모든 경로에 봇의 접근을 허락하지 않는 경우
User-agent: *
Disallow: /
Add Comment
Please,
Sign In
to add comment
Public Pastes
🤑 G2A.com Free Gift Card Guide May 2024 FIX 🎁
GetText | 12 min ago | 0.38 KB
🤑 G2A.com Free Gift Card Guide May 2024 FIX 🤑
GetText | 12 min ago | 0.39 KB
Dragonvale Plant Farming AHK Script No Loggin...
Autohotkey | 49 min ago | 1.63 KB
Dragonvale Plant Farming AHK Script
Autohotkey | 1 hour ago | 3.62 KB
Track Time rev_04
Arduino | 1 hour ago | 3.77 KB
Track Time rev_03
Arduino | 1 hour ago | 3.19 KB
Track Time rev_02
Arduino | 1 hour ago | 2.86 KB
date_utility
Python | 1 hour ago | 3.34 KB
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the
Cookies Policy
.
OK, I Understand
Not a member of Pastebin yet?
Sign Up
, it unlocks many cool features!