MooingCat

MooingCat's Terrible ForgeDB "web scraping" script

Feb 10th, 2021
1,205
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
  1. //Step 1: Go to your world on https://foestats.com/ > Open "Players" > Change number of entries to 100
  2. //Step 2: Open developer tools and go to console (CTRL+SHIFT+I, F12, Right click + inspect (anywhere on page)).
  3. //Step 3: Paste following and hit enter:
  4.  
  5. var output = "";
  6.  
  7. //Step 4: Paste following and hit enter. This will store all players on that page in the "output" variable, and open the next page automatically. Wait a second, and do this again ("up" + "enter" in console). Do this until you've gone through as many pages as you want. I recommend "saving" (see Step 5/6) every now and then in case something goes wrong.
  8.  
  9. for(var i = 0; i<50; i++){
  10. var x = document.getElementsByClassName("odd")[i].cells[2].innerHTML
  11. output = output + x.substring(x.indexOf("\">")+2,x.indexOf("</a>")) + "," + x.substring(x.indexOf("id=")+3,x.indexOf("\">")) + "," + parseInt(document.getElementsByClassName("odd")[i].cells[4].innerText.replace(/,/g,'')) + "," + parseInt(document.getElementsByClassName("odd")[i].cells[5].innerText.replace(/,/g,'')) + "\n";
  12. x = document.getElementsByClassName("even")[i].cells[2].innerHTML
  13. output = output + x.substring(x.indexOf("\">")+2,x.indexOf("</a>")) + "," + x.substring(x.indexOf("id=")+3,x.indexOf("\">")) + "," + parseInt(document.getElementsByClassName("even")[i].cells[4].innerText.replace(/,/g,'')) + "," + parseInt(document.getElementsByClassName("even")[i].cells[5].innerText.replace(/,/g,'')) + "\n";
  14. }
  15. document.getElementsByClassName("paginate_button next")[0].click()
  16.  
  17. //Step 5: When you have gone through all the pages you want, use the following code, which will copy the output to your clipboard:
  18.  
  19. copy(output);
  20.  
  21. //Step 6: Paste the data into a spreadsheet/document. I recommend google sheets, when you paste you can choose to separate the pasted data, this will create a column for each value (player, id, points, fights)
  22.  
  23. //Step 7: Repeat! If you want to start over from a certain page, go to that page and start at Step 3 again. If you want to add more players to your current output, jump to Step 4 instead.
  24.  
  25. //Note: When I did this myself it didn't always work properly, not quite sure why. What I did was that I set up a macro to do "up" + "enter" for all the tabs I needed, then I went through the result manually and updated the sections that were incomplete.
Advertisement
Add Comment
Please, Sign In to add comment