[Solved] HTML scraping with VBA

Having issues accessing new HTML generated after page load.

Solution
Okay, so let me clear a few things up. Using the approach below seems to work fine in most cases with dynamic HTML. However, you may need to wait a few seconds.

Dim ie As InternetExplorer
Set ie = New InternetExplorer
Set x = ie.document.getElementById("foo")

So now to restate the issue that I encountered, which was not being able to access new HTML, was because it was inside an iframe. The solution is below.

Set contentFrame = ie.document.getElementById("foo_frame")
Set x = contentFrame.contentWindow.document.getElementById("foo_obj")

What you could do is call the AJAX directly. That way you get the updates at whatever interval you wish. Its been over two years since I've used VBA and even then I wasn't good at it, but if you can give me more details maybe I can give some useful insights. What site are you trying to scrape data from? What data are you trying to scrape exactly?

I'm trying to scrape the data from a website that I use at work. I can't disclose the site url, but it's an https site if that makes any difference. Basically I want to be able to access the DOM tree so that I can pull data from specific elements, then evaluate it, and then override the existing data with new data. For example, extract the text from an input element, do something with that text, and then assign a new value to that input element. The reason why I'm asking is because after many hours of searching online I finally came across a post in another forum stating that only the original HTML sent from the server is accessible, any HTML added after the fact is not. To clarify, I understand that I might be able to intercept the new HTML via an AJAX call, but it's not being added to the DOM tree for the IE object that I created and therefore I'm still stuck.

What I'm saying is that you might not want to be doing scrapping at all. Its possible that you can call the AJAX directly (ie, call the API endpoint directly) instead, but its just a guess.

Do you mean to send data to the server directly?

Yea, sort of.