Bot traffic and invalid activity are persistent realities of web measurement. Automated requests can originate from search crawlers, monitoring services, scrapers, referral spam, click-fraud operations or coordinated botnet activity. Analytics platforms provide baseline controls for identifying and excluding some of these sources.

The value of analytics data depends on its accuracy, consistency, and governance. Platform filtering can be an appropriate baseline control. It does not, however, remove all forms of invalid activity, nor correct historical data that was affected before controls were in place.

This approach does not solve previously inflated reporting, sophisticated traffic that resembles human behaviour, or the need for periodic review of traffic quality. Filtering should therefore be treated as one control within a wider process of measurement assurance.

1. Categories of invalid traffic

Invalid activity is not a single traffic type. Different sources require different detection signals and may affect analytics, advertising platforms and operational reporting in different ways.

Simple bots and crawlers

Simple bots generally identify themselves through a recognisable user agent or originate from known infrastructure. Search engine crawlers, uptime monitors, link checkers and other automated services may generate requests that appear in server logs and, depending on the implementation, may reach analytics collection points.

Some of this activity is legitimate. Search engine crawling may support discoverability, while monitoring services may be part of the organisation’s operational controls. The issue is usually not that the request exists, but that it is interpreted as a human visit in a business report.

Scrapers and content collectors

Scrapers retrieve pages or assets at scale, often to copy content, monitor prices or build datasets. A scraper may use a recognisable user agent, but it may also imitate a conventional browser and rotate IP addresses.

Where JavaScript execution is limited, scrapers may not create standard browser analytics sessions. Where a headless browser or browser automation is used, the activity can generate page views, events and conversions that contaminate measurement.

Spam referrals and fake direct visits

Referral spam may introduce irrelevant domains into acquisition reports. Some requests are sent directly to analytics endpoints rather than through the website, which can make the activity appear as a referral or direct visit without a corresponding page interaction.

This category can be particularly misleading when acquisition reports are used to assess campaign performance, organic visibility or partner traffic.

Click fraud and paid-channel invalid activity

Click fraud may involve automated or incentivised interactions with paid media. The resulting sessions may have short durations, repeated landing-page views or unusually high proportions of non-converting visits. More sophisticated activity may generate engagement events or follow a plausible navigation path.

The impact may extend beyond analytics. Invalid clicks can influence advertising platform reporting, campaign optimisation, budget allocation and conversion modelling. Analytics data alone may not establish that a click was fraudulent, but it can provide evidence for comparison with advertising and server-side records.

Botnet surges and coordinated attacks

Botnets can generate short, high-volume surges from many IP addresses, devices or geographic locations. The activity may target a website, a form, an API, or an analytics collection endpoint.

A botnet surge is often distinguished by its scale and timing. Requests may increase sharply over a short period, with common page paths, repeated event sequences or unusual concentrations in hosting-provider traffic. A distributed source pattern can make IP-based filtering less effective.

2. Platform-default filtering

Analytics platforms generally provide a baseline method for excluding known bots. These controls can remove a useful proportion of routine machine traffic, but the scope and visibility of the filtering vary by platform.

Google Analytics properties automatically exclude traffic from known bots and spiders. Google states that the mechanism uses its own research and the International Spiders and Bots List maintained by the Interactive Advertising Bureau. The exclusion cannot be disabled, and the volume excluded is not exposed as a standard reportable metric in the property (Google Analytics: Known bot traffic exclusion).

This is useful for known and declared crawlers. It is less likely to identify a headless browser, residential-proxy network or automated user journey deliberately designed to resemble a person. The absence of a visible bot count also means that a reduction in reported activity cannot, by itself, demonstrate the volume that has been removed.

Adobe Analytics provides configurable bot rules. Organisations can enable IAB bot filtering and add custom rules based on user agents, IP addresses, IP ranges or uploaded bot files. Matching activity is excluded from the report suite rather than being corrected retrospectively (Adobe Analytics: Understand and configure bot rules).

Matomo also excludes requests identified as bots or crawlers by default. Its documentation describes additional segments for identifying suspicious behaviour, as well as tracking spam prevention and user-agent exclusions (Matomo: Create segments to identify bot traffic).

The common limitation is that platform defaults are classification controls, not a complete invalid-activity programme. Their effectiveness depends on the quality of the maintained lists, the data available at collection time and the extent to which suspicious traffic differs from normal behaviour.

Recommended action:

• Record which platform-level bot controls are enabled for each property, report suite or site.
• Document whether filtered activity is excluded, segregated or still available for diagnostic reporting.
• Review whether the control applies before data ingestion or only during report processing.
• Treat platform defaults as a baseline rather than as evidence that all non-human activity has been removed.

3. Custom filters and their trade-offs

Custom detection can identify activity outside standard platform lists. Implement it cautiously, because false positives can remove legitimate customers, internal users, accessibility tools, monitoring services, or partner traffic.

User-agent rules

User-agent matching is a practical starting point for clearly identified bots. Terms such as bot, spider, crawler, headless or scanner may reveal routine automated activity. However, user agents are easy to alter and should not be treated as definitive evidence.

A rule based on a broad keyword pattern may also capture legitimate tools. Changes to browser libraries, monitoring products or security scanners can make an existing rule incomplete or over-inclusive.

IP addresses and ranges

IP exclusion can be suitable for known internal networks, data-centre ranges or recurring sources of unwanted requests. It may be less effective for distributed botnets, cloud-hosted automation or residential proxies.

IP addresses can also be shared by multiple users. Blocking an entire range may remove valid customer activity, particularly where mobile networks, corporate gateways or large cloud providers are involved.

Behavioural thresholds

Behavioural signals can identify traffic that does not match ordinary user journeys. Examples include:

• A high number of page views within a very short period.
• Repeated sessions with zero engagement and identical navigation sequences.
• Large numbers of events with no corresponding server-side action.
• Improbable session-duration distributions.
• Unusual combinations of browser, device, language and geography.
• Many visits using an unknown browser or resolution.
• A high action count alongside a 100% bounce rate.

These signals are more adaptable than static lists, but they should be evaluated against normal site behaviour. A product comparison tool, search interface or single-page application may legitimately create high event volumes or short sessions.

Server-side exclusion

Server-side controls can prevent known unwanted requests from reaching analytics collection endpoints. They may also add classification fields, such as a bot-risk flag, before data is sent to a platform.

This approach can improve control over ingestion and reduce the risk that invalid requests affect reporting. It requires reliable access to request metadata, stable ownership between engineering and analytics teams, and testing against legitimate traffic. A server-side filter that is not documented can become difficult to audit when infrastructure or consent controls change.

Recommended action:

• Use multiple signals before classifying activity as invalid.
• Separate high-confidence exclusions from lower-confidence observations.
• Test custom rules against known legitimate monitoring, accessibility and partner traffic.
• Record the reason, owner and review date for every exclusion rule.
• Avoid relying on a single user agent, IP address or engagement threshold.

4. Detecting contamination in reports

The first indication of bot contamination is often a change in a familiar pattern rather than an obvious error. Review should compare affected metrics with historical baselines, server logs, advertising platforms and operational events.

Anomalous traffic spikes

A sudden increase in sessions, page views or events may be associated with a campaign, product launch, technical incident or media mention. If no corresponding business event exists, the increase warrants investigation.

The timing can be informative. A sharp rise outside normal operating hours, particularly when repeated at regular intervals, may indicate automation. A surge restricted to one landing page, host name, event or collection endpoint may also suggest non-human activity.

Unrealistic engagement metrics

Bot activity may produce an unusual combination of metrics, such as very high sessions with negligible engagement, or unusually high events without a corresponding increase in conversions. Neither pattern is conclusive, but both can justify segmentation.

In Matomo, official guidance recommends reviewing combinations such as high actions per visit with very short duration, unknown browser or resolution, and other indicators associated with automated visits (Matomo: Create segments to identify bot traffic).

Session duration patterns

Human session durations generally show a distribution across short, medium and longer visits. Automated activity may create an excess of sessions at the same duration, sessions ending at a repeated interval, or many visits with a duration of zero.

Session-duration data is affected by measurement design. A single-page journey, consent refusal or blocked event may produce a short session even when the visit was genuine. Duration should therefore be assessed alongside page paths, event sequences and server-side evidence.

Dimensions associated with contamination

Useful dimensions for investigation may include:

• Source and medium.
• Campaign and paid-media identifiers.
• Landing page and page path.
• Country, region and city.
• Browser, operating system and device category.
• Host name and data stream.
• User agent, where available.
• IP range or hosting-provider classification in server logs.
• Consent state and collection endpoint.

Recommended action:

• Establish a baseline for traffic volume, engagement and conversion rates before assessing an anomaly.
• Compare analytics activity with server requests, orders, leads and advertising-platform clicks.
• Segment the affected period by source, landing page, device, geography and user agent.
• Preserve the original report exports and investigation notes.
• Avoid declaring traffic invalid solely because a single metric appears unusual.

5. Click fraud and paid-channel impacts

Invalid activity can distort paid-channel analysis in several ways. Automated clicks may increase reported sessions without creating meaningful engagement. Repeated interactions may alter cost-per-session, landing-page engagement and assisted-conversion reporting. If conversion events are also generated, campaign optimisation systems may receive misleading signals.

The effect is not confined to one analytics platform. Advertising platforms may use their own click-validation processes, while analytics platforms may apply different attribution, identity and filtering rules. As a result, a discrepancy between ad clicks and analytics sessions does not automatically indicate fraud.

A structured comparison should consider:

1. The date and time of the activity.
2. Campaign, ad group, keyword or placement identifiers.
3. Clicks, sessions, engaged sessions and conversions.
4. Geographic and device concentrations.
5. Repeated IP or user-agent patterns in server logs.
6. Post-click behaviour and downstream business outcomes.
7. Whether consent or tracking prevention affected analytics collection.

Bot traffic can also create false confidence in a campaign if it generates superficially positive events. Events such as scrolls, video starts or form interactions should be compared with server-confirmed outcomes where possible. A client-side event without a corresponding transaction, lead or application record may be an indicator for further review, not proof of invalidity.

A tagging implementation review can help establish whether the discrepancy originates from traffic quality, duplicate tagging, attribution configuration or incomplete conversion measurement. These causes can appear similar in channel reports but require different remediation.

Recommended action:

• Reconcile advertising clicks, analytics sessions and server-side outcomes by time period and campaign.
• Isolate campaigns with unusual click-to-session or session-to-conversion patterns.
• Share evidence with advertising-platform support where invalid clicks are suspected.
• Avoid changing attribution settings solely to compensate for unexplained traffic.
• Record whether the affected data is unsuitable for optimisation, financial reporting or both.

6. Annotating versus removing affected data

The decision to remove or annotate depends on the purpose of the dataset, the certainty of the classification and the capabilities of the platform.

Removing activity

Pre-ingestion exclusion is generally appropriate where the activity is known to be invalid and the risk of false positives is low. Examples may include a confirmed internal monitoring service, a documented test environment or a known malicious collection pattern.

Removing data can improve the clarity of operational dashboards and reduce the risk of repeated use in routine reporting. It does not, however, repair historical reports that were already inflated. Some platforms also provide limited ability to reconstruct a clean historical dataset after exclusion rules are introduced.

Annotating activity

Annotation is more suitable where classification is uncertain or where the affected period has analytical value. A report may retain the original figures while adding a note such as:

• Suspected automated traffic detected between specified times.
• Paid-channel data under review for possible click fraud.
• Reporting affected by a botnet surge from multiple hosting providers.
• Conversion metrics excluded from optimisation pending validation.
• Historical figures retained for transparency but unsuitable for trend comparison.

Annotations should identify the affected dates, metrics, suspected source, confidence level, investigation owner and remediation status. This is preferable to silently changing a published number without preserving the reason.

Creating a clean analytical view

Where the platform permits it, a clean segment or derived dataset can be used alongside the unaltered source. This supports comparison between reported activity and the organisation’s current assessment of valid activity.

The distinction between source data, filtered reporting and corrected analysis should remain clear. It is generally preferable to preserve the original evidence and document the transformation applied to downstream reporting.

Recommended action:

• Define thresholds for exclusion, annotation and escalation before the next incident.
• Preserve original exports, logs and platform settings for the affected period.
• Add annotations to dashboards and recurring reports where trend interpretation may be affected.
• Mark corrected datasets with a version, owner and methodology.
• Do not present reconstructed historical figures as if they were platform-native measurements.

7. Ongoing monitoring and alerting

Bot contamination is more manageable when detection is operational rather than occasional. Monitoring should combine volume, quality and technical indicators.

Useful alerts may include:

• Session or event volume exceeding a historical threshold.
• Traffic increasing without a corresponding change in server requests or business outcomes.
• A sudden concentration in one country, network, hostname or landing page.
• Engagement rates falling outside an established range.
• A high number of events from unknown browsers or resolutions.
• A material difference between advertising clicks and analytics sessions.
• Analytics collection requests increasing without a similar rise in page delivery.

Alert thresholds should account for normal seasonality, campaigns, launches and operational incidents. Fixed thresholds may generate excessive noise for organisations with substantial fluctuations. Relative thresholds, peer-group comparisons and anomaly detection may be more appropriate where sufficient history exists.

Matomo documents tracking spam prevention and recommends investigating unexpected referrers, user agents, IP addresses and unusual activity before configuring exclusions (Matomo: Block spam and bot traffic with Tracking Spam Prevention). Similar principles can be applied across platforms, even where the implementation details differ.

Operational ownership should be explicit. Analytics teams may own report interpretation, engineering teams may own server-side controls, paid-media teams may own advertising-platform disputes, and privacy or security teams may need to review data-handling implications.

A broader analytics data quality process can incorporate bot monitoring with tag validation, discrepancy analysis and release governance.

8. When automated filtering is likely to be sufficient

Automated filtering is likely to be sufficient where:

• The activity is generated by recognised search crawlers or routine monitoring services.
• Platform defaults are enabled and the traffic pattern is low volume and stable.
• The source uses a consistent, identifiable user agent or IP range.
• No material paid-media, conversion or financial reporting decision depends on the affected traffic.
• Historical reports do not show a significant unexplained change.
• The organisation has a documented review process for false positives and rule changes.

Additional review is necessary when traffic is distributed across many IP addresses, imitates human navigation, affects paid channels or coincides with material changes in conversions, revenue or campaign performance. Automated filtering may also be insufficient where the data has already been used in business decisions and historical correction is required.

9. A practical approach

A controlled process can be organised into the following stages.

1. Define the reporting purpose

Specify whether the affected data is used for operational dashboards, campaign optimisation, financial reporting, product analysis, forecasting or regulatory evidence. The required level of certainty will differ by purpose.

2. Establish a baseline

Record typical traffic volumes, engagement distributions, conversion rates, channel ratios and technical dimensions. Baselines should cover comparable weekdays, seasonal periods and campaign conditions.

3. Confirm platform controls

Review known-bot settings, IAB-based rules, custom exclusions, internal-traffic filters, consent configuration and server-side collection controls. Record when each control became active.

4. Isolate the anomaly

Segment by time, source, campaign, landing page, device, geography, hostname, event, and user agent. Compare the segment with server logs, orders, leads and advertising-platform records.

5. Classify confidence

Use categories such as confirmed invalid, probable invalid, under review and likely legitimate. A confidence rating prevents uncertain hypotheses from becoming permanent exclusions.

6. Select the response

Choose between pre-ingestion exclusion, platform filtering, a reporting segment, historical annotation, retrospective correction or escalation to a media or security provider.

7. Test for false positives

Run the proposed rule against known users, monitoring services, partners, internal networks and important customer journeys. Check whether legitimate conversion paths are affected.

8. Implement with ownership

Deploy the control through the appropriate tag management, analytics, application or infrastructure process. Assign an owner, change reference, review date and rollback method.

9. Validate after release

Compare traffic, engagement, conversion and discrepancy metrics before and after implementation. Confirm that the rule has reduced the targeted pattern without creating a new data gap.

10. Document and annotate

Update the measurement specification, dashboard notes, incident record and governance documentation. Historical periods should remain identifiable even where a clean analytical view has been created.

Conclusion

Bot filtering is a useful baseline control, but it does not solve the full problem of invalid activity. It does not restore data inflated by earlier attacks, identify every automated session that imitates human behaviour, or replace review of paid-channel and conversion discrepancies.

A reliable approach depends on clear specifications, cautious rule design, testing against legitimate journeys and documented ownership. Where historical data is affected, annotation may be more defensible than silent removal, particularly when the classification is uncertain.

The objective is not to claim that every anomalous request has been identified. It is to ensure that the organisation can distinguish platform-filtered data, suspected invalid activity and validated business performance, with sufficient evidence to support each reporting decision.

If you’d like to hear more on this topic and set up a 1:1 call with us, please use the contact form. Contact a Consultant