@adguard/tsurlfilter

This is a TypeScript library that implements AdGuard's content blocking rules

Usage no npm install needed!

<script type="module">
  import adguardTsurlfilter from 'https://cdn.skypack.dev/@adguard/tsurlfilter';
</script>

README

TSUrlFilter

NPM

This is a TypeScript library that implements AdGuard's content blocking rules.

Idea

The idea is to have a single library that we can reuse for the following tasks:

Usage

Install the tsurlfilter:

npm install @adguard/tsurlfilter

API description

Public classes

Engine

Engine is a main class of this library. It represents the filtering functionality for loaded rules

Constructor
    /**
     * Creates an instance of Engine
     * Parses filtering rules and creates a filtering engine of them
     *
     * @param ruleStorage storage
     * @param configuration optional configuration
     *
     * @throws
     */
    constructor(ruleStorage: RuleStorage, configuration?: IConfiguration | undefined)
matchRequest
    /**
     * Matches the specified request against the filtering engine and returns the matching result.
     *
     * @param request - request to check
     * @return matching result
     */
    matchRequest(request: Request): MatchingResult
getCosmeticResult
    /**
     * Gets cosmetic result for the specified hostname and cosmetic options
     *
     * @param request - request to check
     * @param option mask of enabled cosmetic types
     * @return cosmetic result
     */
    getCosmeticResult(request: Request, option: CosmeticOption): CosmeticResult
Starting engine
    const list = new StringRuleList(listId, rulesText, false);
    const ruleStorage = new RuleStorage([list]);

    const config = {
        engine: 'extension',
        version: '1.0.0',
        verbose: true,
    };

    setConfiguration(config)

    const engine = new Engine(ruleStorage);
Matching requests
    const request = new Request(url, sourceUrl, RequestType.Document);
    const result = engine.matchRequest(request);
Retrieving cosmetic data
    const cosmeticResult = engine.getCosmeticResult(request, CosmeticOption.CosmeticOptionAll);

MatchingResult

MatchingResult contains all the rules matching a web request, and provides methods that define how a web request should be processed

getBasicResult
    /**
     * GetBasicResult returns a rule that should be applied to the web request.
     * Possible outcomes are:
     * returns null -- bypass the request.
     * returns a whitelist rule -- bypass the request.
     * returns a blocking rule -- block the request.
     *
     * @return basic result rule
     */
    getBasicResult(): NetworkRule | null
getCosmeticOption

This flag should be used for getCosmeticResult(request: Request, option: CosmeticOption)

    /**
     * Returns a bit-flag with the list of cosmetic options
     *
     * @return {CosmeticOption} mask
     */
    getCosmeticOption(): CosmeticOption
Other rules
    /**
     * Return an array of replace rules
     */
    getReplaceRules(): NetworkRule[]

    /**
     * Returns an array of csp rules
     */
    getCspRules(): NetworkRule[]

    /**
     * Returns an array of cookie rules
     */
    getCookieRules(): NetworkRule[]

CosmeticResult

Cosmetic result is the representation of matching cosmetic rules. It contains the following properties:

    /**
     * Storage of element hiding rules
     */
    public elementHiding: CosmeticStylesResult;

    /**
     * Storage of CSS rules
     */
    public CSS: CosmeticStylesResult;

    /**
     * Storage of JS rules
     */
    public JS: CosmeticScriptsResult;

    /**
     * Storage of Html filtering rules
     */
    public Html: CosmeticHtmlResult;

    /**
     * Script rules
     */
    public getScriptRules(): CosmeticRule[];
Applying cosmetic result - css
   const css = [...cosmeticResult.elementHiding.generic, ...cosmeticResult.elementHiding.specific]
           .map((rule) => `${rule.getContent()} { display: none!important; }`);

    const styleText = css.join('\n');
    const injectDetails = {
        code: styleText,
        runAt: 'document_start',
    };

    chrome.tabs.insertCSS(tabId, injectDetails);
Applying cosmetic result - scripts
    const cosmeticRules = cosmeticResult.getScriptRules();
    const scriptsCode = cosmeticRules.map((x) => x.getScript()).join('\r\n');
    const toExecute = buildScriptText(scriptsCode);

    chrome.tabs.executeScript(tabId, {
        code: toExecute,
    });

DnsEngine

DNSEngine combines host rules and network rules and is supposed to quickly find matching rules for hostnames.

Constructor
    /**
     * Builds an instance of dns engine
     *
     * @param storage
     */
    constructor(storage: RuleStorage)
match
    /**
     * Match searches over all filtering and host rules loaded to the engine
     *
     * @param hostname to check
     * @return dns result object
     */
    public match(hostname: string): DnsResult
Matching hostname
    const dnsResult = dnsEngine.match(hostname);
    if (dnsResult.basicRule && !dnsResult.basicRule.isWhitelist()) {
        // blocking rule found
        ..
    }

    if (dnsResult.hostRules.length > 0) {
        // hosts rules found
        ..
    }

RuleConverter

Before saving downloaded text with rules it could be useful to run converter on it. The text will be processed line by line, converting each line from known external format to Adguard syntax.

convertRules
    /**
     * Converts rules text
     *
     * @param rulesText
     */
    public static convertRules(rulesText: string): string {

ContentFiltering

Content filtering module, it applies html-filtering and $replace rules. The rules could be retrieved with parsing the result of engine.matchRequest.

Constructor
    /**
     * Creates an instance of content filtering module
     *
     * @param filteringLog
     */
    constructor(filteringLog: FilteringLog)
apply
    /**
     * Applies content and replace rules to the request
     *
     * @param streamFilter stream filter implementation
     * @param request
     * @param contentType Content-Type header
     * @param replaceRules array of replace rules
     * @param htmlRules array of html-filtering rules
     */
    public apply(
        streamFilter: StreamFilter,
        request: Request,
        contentType: string,
        replaceRules: NetworkRule[],
        htmlRules: CosmeticRule[],
    ): void
Applying content-filtering rules
    const request = new Request(url, sourceUrl, requestType);
    request.requestId = requestId;
    request.tabId = tabId;
    request.statusCode = statusCode;
    request.method = method;

    contentFiltering.apply(
        chrome.webRequest.filterResponseData(requestId),
        request,
        'text/html; charset=utf-8',
        replaceRules,
        htmlRules,
    );

StealthService

Stealth service module, it provides some special functionality like removing tracking parameters and cookie modifications

Constructor
    /**
     * Constructor
     *
     * @param config
     */
    constructor(config: StealthConfig)
Stealth configuration
    const stealthConfig = {
        stripTrackingParameters: true,
        trackingParameters: 'utm_source,utm_medium,utm_term',
        selfDestructThirdPartyCookies: true,
        selfDestructThirdPartyCookiesTime: 0,
        selfDestructFirstPartyCookies: true,
        selfDestructFirstPartyCookiesTime: 1,
    };

    this.stealthService = new StealthService(stealthConfig);
removeTrackersFromUrl
    /**
     * Strips out the tracking codes/parameters from a URL and return the cleansed URL
     *
     * @param request
     */
    public removeTrackersFromUrl(request: Request): string | null
getCookieRules
    /**
     * Returns synthetic set of rules matching the specified request
     */
    public getCookieRules(request: Request): NetworkRule[]

RedirectsService

Redirects service module applies $redirect rules. More details on sample extension.

Init service
    const redirectsService = new RedirectsService();
    await redirectsService.init();
Usage
    const result = engine.matchRequest(request);
    const requestRule = result.getBasicResult();

    if (requestRule.isOptionEnabled(NetworkRuleOption.Redirect)) {
        const redirectUrl = redirectsService.createRedirectUrl(requestRule.getAdvancedModifierValue());
    }

CookieFiltering

Cookie filtering module applies $cookie rules. Adds a listener for CookieApi.setOnChangedListener(..) then applies rules from RulesFinder to event cookie.

Constructor

Check CookieApi and RulesFinder interfaces

    /**
     * Constructor
     *
     * @param cookieManager
     * @param filteringLog
     * @param rulesFinder
     */
    constructor(cookieManager: CookieApi, filteringLog: FilteringLog, rulesFinder: RulesFinder)
Public methods
    /**
     * Parses response header set-cookie.
     * Saves cookie third-party flag
     *
     * @param request
     * @param responseHeaders Response headers
     */
    processResponseHeaders(request: Request, responseHeaders: Header[]): void;

    /**
     * Filters blocking first-party rules
     *
     * @param rules
     */
    getBlockingRules(rules: NetworkRule[]): NetworkRule[];

RuleValidator

This module is not used in the engine directly, but it can be used to validate filter rules in other libraries or tools

Public methods
    /**
     * Validates raw rule string
     * @param rawRule
     */
    public static validate(rawRule: string): ValidationResult
    /**
    * Valid true - means that the rule is valid, otherwise rule is not valid
    * If rule is not valid, reason is returned in the error field
    */
    interface ValidationResult {
        valid: boolean;
        error: string | null;
    }

RuleSyntaxUtils

This module is not used in the engine directly, but it can be used in other libraries

Public methods
    /**
     * Checks if rule can be matched by domain
     * @param ruleText
     * @param domain
     */
    public static isRuleForDomain(ruleText: string, domain: string): boolean {
    /**
     * Checks if rule can be matched by url
     * @param ruleText
     * @param url
     */
    public static isRuleForUrl(ruleText: string, url: string): boolean {

Content script classes

Classes provided for page context:

CssHitsCounter

Class represents collecting css style hits process.

Initialization:
    const cssHitsCounter = new CssHitsCounter((stats) => {
        chrome.runtime.sendMessage({type: "saveCssHitStats", stats: JSON.stringify(stats)});
    });

CookieController

This class applies cookie rules in page context

Usage:
    const rulesData = rules.map((rule) => {
        return {
            ruleText: rule.getText(),
            match: rule.getAdvancedModifierValue(),
        };
    });

    const cookieController = new CookieController((rule) => {
        console.debug('On cookie rule applied');
    });

    cookieController.apply(rulesData);

Sample extension

Source code of the sample extension is located in the directory ./sample-extension

To build sample extension run in the root directory

npm run build-extension

This command builds extension to ./dist-extension directory. After that it's ready to be added to Chrome using "Load unpacked" in developer mode.

To test if this extension works correctly you can use next test pages:

Test pages:

Development

NPM scripts

  • npm t: Run test suite
  • npm start: Run npm run build in watch mode
  • npm run test:watch: Run test suite in interactive watch mode
  • npm run test:prod: Run linting and generate coverage
  • npm run test:benchmarks: Run benchmark tests, inspect in chrome://inspect
  • npm run build: Generate bundles and typings, create docs
  • npm run lint: Lints code
  • npm run commit: Commit using conventional commit style (husky will tell you to use it if you haven't :wink:)
  • npm run build-extension: Build sample chrome extension

Excluding peerDependencies

On library development, one might want to set some peer dependencies, and thus remove those from the final bundle. You can see in Rollup docs how to do that.

Good news: the setup is here for you, you must only include the dependency name in external property within rollup.config.js. For example, if you want to exclude lodash, just write there external: ['lodash'].

Git Hooks

There is already set a precommit hook for formatting your code with Eslint :nail_care: