Click here to Skip to main content
15,886,110 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
I've written an html scraper which runs fine apart from on pages that have a popup message box (example here Untitled hosted at ImgBB — ImgBB[^]) , these seems to remain open, which eventually causes the machine to run out of memory and crash.

I've tried sending a keypress "Enter" but that doesn't seem to work.

Any ideas much appreciated, thanks.

What I have tried:

C#
var browser = await Puppeteer.LaunchAsync(new LaunchOptions
{
    IgnoreDefaultArgs = true,
    Args = new[] { "--disable-extensions" },
    Headless = false,
    IgnoreHTTPSErrors = true,
    Timeout = 0
});

using (var page = await browser.NewPageAsync())
{
    try
    {
        await page.GoToAsync(url, timeout: 60000);
        await page.Keyboard.PressAsync("Enter"); //clear pop ups
        await Task.Delay(10000);

        //get html
        var content = await page.GetContentAsync();

        //get screenshot
        Stream _stream = await page.ScreenshotStreamAsync(new ScreenshotOptions { FullPage = true });
        MemoryStream ms = new MemoryStream();
        await _stream.CopyToAsync(ms);
        fs.StoreScreenshot(ms, $"{RecordId}.jpg");

        await page.CloseAsync();
        await browser.CloseAsync();
        }
        return content;
    }
    catch (Exception ex)
    {
        log.Error($"Cannot load page, {ex.Message}");
        await page.CloseAsync();
        await browser.CloseAsync();
        return null;
    }
}
Posted
Comments
[no name] 12-May-23 11:31am    
Key presses only go to the element that has focus.
punk_legend 16-May-23 16:58pm    
Good point, hadn't thought of that. I assume it's going to be very hard to pass focus to the popup as dependant on the website they will have different names? Any ideas how I can close/prevent these popups from appearing?

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900