Hi everyone,
As the title states, I want to automate the task of searching on the internet for a list of keywords, on certain webpages contained in another list. The goal is to have this automated script notify me via email if the automated search returned any positive results, and in which of the webpages it found the result. It would be an added bonus is the script could be made to run automatically everyday ( via windows automated tasks or something of the sort)
I am absolutely out of my depth but I am almost certain that what I’m thinking of can be done with some sort of script. If any of you find folks can point me in the right direction regarding what programming language would be the easiest to get it running and managing it ( ie adding new keywords or sites), and also any good tutorials for writing the code.
If I interpreted your post correctly, it sounds like you want to search a set of known URLs, rather than crawling.
Since you’re on windows, assuming PowerShell for scripting language, but these generic steps could easily be done in Python or what ever you prefer, some API will be different.
add urls to a plain text file, separated by newlines urls.txt
add keywords to a plain text file, separated by newlines keywords.txt
Files don’t have to be new lines. Could use csv (comma separated values) or what ever format you prefer.
Breaking it down into steps for a script might look something like:
Get-Content to load the 2 files, paths passed in as arguments to the script
iterate the urls list
curl or Invoke-WebRequest to retrieve the web content for each url
iterate each keyword
Select-String to search the content for the keyword
if matches are found, Send-MailMessage for the email
You may have to set certain headers for some sites play nice. (hint: just match your browser’s headers and UA in the web request)
Task Scheduler is what you want for the automated daily running.
Maintaining it is as simple as adding urls and keywords to the files.
As far as tutorials, youtube is a great resource. If you’re new to programming, start with loading the text file and printing all the url’s from the text file. Learn what a variable, a for loop and an if statement are. That should get you started.
Feel free to ask questions if you get stuck.
If you don’t want to DIY, you may find a tool here that does what you need:
Thanks for the feedback. Yes your interpretation of the question is on point!
Just to try it the DIY route, works you recommend python as the most noob-friendly language? Or do you think it works be worth trying to code on an alternate language?
Yes I would recommend Python, but not because “it’s the most noob friendly”. It’s a good language, and perfectly suited for a task like this. It also happens have a simple/clean syntax that is “noob friendly” as a bonus. The added benefit to Python is it is OS-agnostic if you want to run this script on Linux instead later. And tons of libraries to leverage.
Powershell and Python are both simple. I gave you the high level break down for PoweShell because it’s native to windows and already there. You can open the PowerShell ISE built in editor if you want a sort of IDE experience. If you decide Python, PyCharm and VSCodium are both nice editors.
Learn the fundamentals of programming and the language doesn’t really matter.