@Wendell recently mentioned in one of the tek episodes that he has bots that generates web traffic clicking though stuff and others that monitor things like prices of tickets for shows.
I would love to know what language these were written in, What all of the little bots he has do (or can do) and maybe some videos on the how they work and maybe some guides for someone to build something like this of their own?
I would love to learn more about building stuff like this for myself, I have been a sysadm for years, (I started very young) but only recently learned to code when I started my University course, which sadly although has given me a basic understanding of Java hasn't given me a lot.....
I did thumb through to see if a post like this already existed and didn't find anything, If there is I do apologize!.
Yes you can definitely use selenium. Might be a bit difficult though, I tried writing a bit of code with it the last time this topic came up. I couldn't get get sending mouse events to not crash the program for some reason, not sure why though.
He really should turn them into proper programs that you can install and have run at startup (both on Windows and Linux, of course). I reckon a lot of people here would buy it.
Hell, make it a subscription-based model so it can provide a steady income for another intern.
I vaguely recall Wendell answering this question in another recent post: At the time Wendell said the script for the bots were made up of a series of scripts that he had created over the years.
Wendell also stated at this stage he wasn't in a position to release details of his bot-code (the mods would know the post I'm referring to)
Yea, I wouldn't always want to give up some of the unique awesome stuff I had either. I have things (although not this cleaver or cool) that are scripts and things that all work together to do things that wouldn't really be suitable to hand to someone else to use without been re-written (but who has enough time for that in the IT world), but I would love to see some videos on how this kinda thing is made, I know I would get something out of it and I am sure lots of others would
check these tools, they are less insane than trying to do this with selenium. Though I do have some selenium scripts for stuff that has to execute in-browser javascript to login or otherwise use the site. After that phantomjs or phpquery. Or some stuff like this in python that I've cobbled together.
Computers talking to other computers? That's web 3.0! lol
These kinds of things didn't always exist. For example to do the ebay thing where it buys me stuff randomly, I have to do that in phantomjs I bet now because the way I used to do it depends on javascript not being a thing in the web scraping. Which sadly now I think it is.
Web 3.0 is going to happen when there is an api for everything. It'll be like stock traders because even the most basic day traders now do some algorithmic trades. This is just like that but for content-at-large on the internet that is difficult to find with a search engine, or too tedious and a waste of your time to constantly do on your own over and over again.
Imagine being able to ask questions like, show me a list of stories that show up on the reddit front page, and also appear on hacker news. Or show me a list of stories that show up on both reddit/r/technology and in our own forums and have more than 1000 upvotes.
Eventually, you build classes built on tools like these that allow you to do "queries" with selectors almost as simple as jquery/phpquery.
its (was) governed by rules -- it had a budget of $50 and I would rate 0-50 the item when it came in, and it would "learn" but that part of it was shitty.
Later versions would only 'snipe' auctions (no buy it now).. it tried to buy at least 4 things a month and the price would be chosen on a logarithmic scale with $50 for the first buy of the month being very unlikely and $5-$10 items being much more likely. any leftover dollars from one month would go into the next month's pool.
limited to only technology items. limited (almost execlusively) to items with free or $1 shipping. Later I tried to exclude books/manuals but I did score a sweet shrinkwarpped copy of Windows for Workgroups 3.11 for like $5.88 shipped.
Some things I ended up with:
A VAX vacuum cleaner (on which I plan to install OpenVMS) Shrinkwrapped Copy of WFW3.11 50 "antique" shareware DVD Roms a 5.25" CD-ROM Autoloader (it was a CDROM that would hold 5-6 disks).
I also ended up getting a lot of random cables I tried to modify it not to buy cables. I also got a lot of random craps. I think I ended up with a sidewinder gamepad (MS) and a motherboard or two, some dead stuff (had to modify the program to exclude items with dead in the title), also weighted NIB slightly heavier. A fair number of books about obsolete shitty software, most of which I trashed or gave away. I think I still have a copy of Autocad 14, though, which I kept. It came with "Using Autocad 14" which was useful in my early CNC days. At the time, it was only 3-4 versions behind current, but the dongle was really fiddly which I fixed with a bit of soldering.
I can't wait for everything to have an API like that. Speaking of APIs there is usually a library around for some major languages to make interacting with it very easy when they exist. I am starting a few scripts for a simple bot. So far I have some simple code using python, and the twitter4j library in Java that plays a notification sound when I get mentioned on twitter. I've also started a webscraper to look for jobs. I'll share the twitter one below, in case anyone is interested. I could also post the job-scraper code, but so far it has no way of 'understanding' the job posts it finds (on indeed.com).
That sounds awesome, I remember there was an XKCD where he did something similar. Would love to see a video on the OpenVMS install on that VAX! :-P
I have tools that fetch my mail from all my accounts (business and personal) and pass it all though my own mail server where my business mail gets delivered directly, it sorts spam and decides how important the mail is so only important stuff gets to the inbox and anything else gets filed away automatically until I have time to read it but that isn't really all that cleaver.
It really interested me when you had mentioned these bots, I have my own home server (in fact I have a private Linux server at each location I work at for keeping all my tools and stuff on because bandwidth is either really good or really bad in the UK) Can I ask what makes you decide to build a bot to do something?