11/02/2024:

	WARC/1.0
	WARC-Type: request
	WARC-Date: 2024-11-04T23:04:08Z
	WARC-Record-ID: <urn:uuid:9ab260bc-ba72-45a5-9de8-c050896fcbd7>
	Content-Length: 273
	Content-Type: application/http; msgtype=request
	WARC-Warcinfo-ID: <urn:uuid:f3b8b2e1-5f14-4757-bd81-b87f95221b05>
	WARC-IP-Address: 159.69.231.144
	WARC-Target-URI: https://wiki.diasporafoundation.org/robots.txt
	WARC-Protocol: h2
	WARC-Protocol: tls/1.3
	WARC-Cipher-Suite: TLS_AES_256_GCM_SHA384

	GET /robots.txt HTTP/1.1
	User-Agent: CCBot/2.0 (https://commoncrawl.org/faq/)
	Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
	Accept-Language: en-US,en;q=0.5
	Accept-Encoding: br,gzip
	Host: wiki.diasporafoundation.org
	Connection: Keep-Alive



	WARC/1.0
	WARC-Type: response
	WARC-Date: 2024-11-04T23:04:08Z
	WARC-Record-ID: <urn:uuid:c8783a8f-c05b-44c4-a2f2-73adb1587122>
	Content-Length: 289
	Content-Type: application/http; msgtype=response
	WARC-Warcinfo-ID: <urn:uuid:f3b8b2e1-5f14-4757-bd81-b87f95221b05>
	WARC-Concurrent-To: <urn:uuid:9ab260bc-ba72-45a5-9de8-c050896fcbd7>
	WARC-IP-Address: 159.69.231.144
	WARC-Target-URI: https://wiki.diasporafoundation.org/robots.txt
	WARC-Protocol: h2
	WARC-Protocol: tls/1.3
	WARC-Cipher-Suite: TLS_AES_256_GCM_SHA384
	WARC-Payload-Digest: sha1:XL2MLZYFZTANU54S6K77NZBLUXHJKO5P
	WARC-Block-Digest: sha1:WV2WK657ENZNCRNHJCXL2UOXWR7U35XX
	WARC-Identified-Payload-Type: text/x-robots

	HTTP/1.1 200
	server: nginx/1.27.2
	date: Mon, 04 Nov 2024 23:04:08 GMT
	content-type: text/plain
	last-modified: Fri, 13 Sep 2024 18:52:00 GMT
	etag: W/"1c-62204b7e88e25"
	alt-svc: h3=":443", h2=":443"
	X-Crawler-content-encoding: gzip
	Content-Length: 28

	User-agent: *
	Disallow: /w/


Source:
https://index.commoncrawl.org/CC-MAIN-2024-51-index?url=wiki.diasporafoundation.org/robots.txt&output=json

{"urlkey": "org,diasporafoundation,wiki)/robots.txt", "timestamp": "20241210133720", "url": "https://wiki.diasporafoundation.org/robots.txt", "mime": "text/plain", "mime-detected": "text/x-robots", "status": "200", "digest": "XL2MLZYFZTANU54S6K77NZBLUXHJKO5P", "length": "643", "offset": "1255272", "filename": "crawl-data/CC-MAIN-2024-51/segments/1733066061339.24/robotstxt/CC-MAIN-20241210132922-20241210162922-00694.warc.gz"}

12/10/2024:

	WARC/1.0
	WARC-Type: request
	WARC-Date: 2024-12-10T13:37:20Z
	WARC-Record-ID: <urn:uuid:1edfa9b4-959d-4439-a146-8625cf6038cb>
	Content-Length: 273
	Content-Type: application/http; msgtype=request
	WARC-Warcinfo-ID: <urn:uuid:8144e185-abff-422a-9666-a45dad0e2d04>
	WARC-IP-Address: 159.69.231.144
	WARC-Target-URI: https://wiki.diasporafoundation.org/robots.txt
	WARC-Protocol: h2
	WARC-Protocol: tls/1.3
	WARC-Cipher-Suite: TLS_AES_256_GCM_SHA384

	GET /robots.txt HTTP/1.1
	User-Agent: CCBot/2.0 (https://commoncrawl.org/faq/)
	Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
	Accept-Language: en-US,en;q=0.5
	Accept-Encoding: br,gzip
	Host: wiki.diasporafoundation.org
	Connection: Keep-Alive



	WARC/1.0
	WARC-Type: response
	WARC-Date: 2024-12-10T13:37:20Z
	WARC-Record-ID: <urn:uuid:77346579-5141-46f6-8665-c38d0ee07a45>
	Content-Length: 289
	Content-Type: application/http; msgtype=response
	WARC-Warcinfo-ID: <urn:uuid:8144e185-abff-422a-9666-a45dad0e2d04>
	WARC-Concurrent-To: <urn:uuid:1edfa9b4-959d-4439-a146-8625cf6038cb>
	WARC-IP-Address: 159.69.231.144
	WARC-Target-URI: https://wiki.diasporafoundation.org/robots.txt
	WARC-Protocol: h2
	WARC-Protocol: tls/1.3
	WARC-Cipher-Suite: TLS_AES_256_GCM_SHA384
	WARC-Payload-Digest: sha1:XL2MLZYFZTANU54S6K77NZBLUXHJKO5P
	WARC-Block-Digest: sha1:S2AO6CCP5OPMXEC42ZAWBGV3GCHHPBLS
	WARC-Identified-Payload-Type: text/x-robots

	HTTP/1.1 200
	server: nginx/1.27.2
	date: Tue, 10 Dec 2024 13:37:20 GMT
	content-type: text/plain
	last-modified: Fri, 13 Sep 2024 18:52:00 GMT
	etag: W/"1c-62204b7e88e25"
	alt-svc: h3=":443", h2=":443"
	X-Crawler-content-encoding: gzip
	Content-Length: 28

	User-agent: *
	Disallow: /w/


Source:
https://index.commoncrawl.org/CC-MAIN-2024-46-index?url=https://wiki.diasporafoundation.org/robots.txt&output=json

{"urlkey": "org,diasporafoundation,wiki)/robots.txt", "timestamp": "20241104230408", "url": "https://wiki.diasporafoundation.org/robots.txt", "mime": "text/plain", "mime-detected": "text/x-robots", "status": "200", "digest": "XL2MLZYFZTANU54S6K77NZBLUXHJKO5P", "length": "645", "offset": "1265314", "filename": "crawl-data/CC-MAIN-2024-46/segments/1730477027861.84/robotstxt/CC-MAIN-20241104225856-20241105015856-00694.warc.gz"}