Stealthy fetcher mode triggers unwanted ad/analytics requests impacting proxy usage

[rubiagul280]

When loading a page in stealthy fetcher mode, the request unexpectedly triggers dozens of additional calls to ad networks, analytics trackers, and third‑party resources.
Even after disabling resource loading, these ad/analytics requests still occur and inflate proxy usage—sometimes 80–100 extra requests for what should be a single page fetch. This makes proxy bandwidth consumption and request counts much higher than expected.

Question:
How can we effectively block or filter these unwanted third‑party requests so that only the main domain traffic is counted and proxied? Are there recommended approaches (e.g., request interception, domain whitelisting, or proxy‑side filtering) to ensure stealthy fetcher mode doesn’t include ads/analytics calls?

[D4Vinci]

Hi mate, is playing around with load_dom=False and network_idle=True not helping in your case? If that's the case, I would recommend some more advanced solutions till the next release is out.

[rubiagul280]

No, it didn't work for me. Here is the code

async def _fetch_first_page(self, search_url: str) -> Optional[str]:
    """Fetch first page using StealthyFetcher with load_dom=False to prevent JS execution."""
    try:
        logger.info(f"Starting page fetch: {search_url}")
        proxy = self._get_proxy_config()

        logger.info("Calling StealthyFetcher.async_fetch with load_dom=False...")
        response = await StealthyFetcher.async_fetch(
            url=search_url,
            headless=True,
            disable_resources=True,  # Block fonts, images, media, stylesheets, etc.
            load_dom=False,          # Don't wait for JS to execute - prevents tracking scripts
            network_idle=True,       # Wait for network to settle
            timeout=60000,
            wait=2000,
            proxy=proxy,
            # Anti-detection
            google_search=True,
            block_webrtc=True,
            hide_canvas=True,
            allow_webgl=True,
        )

        logger.info(f"Response received: {type(response)}")
        if not response:
            logger.error("Response is None")
            return None

        # Get HTML content from response
        html_content = getattr(response, 'html', None) or getattr(response, 'text', None) or str(response)
        logger.info(f"Got HTML content, length: {len(html_content) if html_content else 0}")

        if not html_content:
            logger.error("No HTML content found")
            return None

        return html_content

    except Exception as e:
        logger.error(f"Error fetching first page: {e}")
        return None

[D4Vinci]

Hey @rubiagul280

Before trying complicated solutions, can you try the beta of the next version and let me know if it resolves this issue for you?
Use this command to install pip install "git+https://github.com/D4Vinci/Scrapling.git@dev#egg=scrapling[all]" -U

This update includes several changes, some of which aim to further optimize browser sessions. Like automatically preventing autoplay media, such as ads, from autoplaying as they do now.

The browser fetchers/sessions, such as StealthyFetcher, now have a new argument called blocked_domains, which takes a set of domain names (strings) and prevents requests to those domains. This will be very helpful with ads.

Checkout the new readme for it here for some explanation: https://github.com/D4Vinci/Scrapling/tree/dev
This version is currently being discussed on the Discord server before release.

[rubiagul280]

@D4Vinci Thanks for the update. I will test it and get back to you

[rubiagul280]

@D4Vinci Integrated the deployed version and it's working well on my end
Thanks for the update

[rubiagul280]

Hi @D4Vinci

I encountered this issue on 3–4 different platforms related to blocked_domains. Whether it was a normal website or a social media platform, blocked_domains only worked for a few domains. Even after adding accounts.google.com to the list, it still appeared in the proxy activity logs, and some other platform-related domains were not blocked either. To resolve this, I used Patchright and blocked the requests in the route handler first, then allowed only the required document and other necessary resources. Can you check this issue again?

[D4Vinci]

Hey @rubiagul280 we are using the route approach as well; I will look at it, but are you using the latest published version? (0.4.2)
I'm even thinking of fully simulating browser addons like uBlock in the upcoming versions.

[Nooruddin87]

Use request interception in tools like Puppeteer or Playwright to whitelist only the target domain and abort anything matching ad/analytics patterns (e.g., Google Analytics, DoubleClick). You can also add proxy-side rules (like in mitmproxy) to block known tracker domains so only first-party traffic goes through and your proxy usage stays clean.