Archive [dot] org is a great resource for viewing historical copies of URLs on a domain. And hey, they even have an API!
<narrators voice>
But what if I told you there's a better way...
</narrators voice>
A lesser known way to get ALL the URLs that Archive [dot] org knows about a domain, along with the:
- url key
- timestamp
- original URL
- mimetype
- status code
and other data points, is to use their TXT output URL.
Wait, what is their TXT output URL?
Well, friends, let me tell you.
In an overly simplified way of saying it, it's a URL that—when you enter a domain as the value of its URL parameter—will output their entire list of known URLs over time, in text format
???? ???? ????
Here's what it looks like:
https://web.archive [dot] org/cdx/search/csx?url=[domain]*&output=txt
And here is a basic javascript that you can save as a bookmark, which will allow you to get the list for any domain you're visiting, opened in a new window, with a simple click of a button ❤️
-----
javascript:(function(){
var domain = window.location.hostname;
var newURL = "https://web.archive [dot] org/cdx/search/csx?url=" + domain + "*&output=txt";
window.open(newURL, '_blank');
})();
(replace [dot] with a period to make this work!)
