Writing an Expression Rule for Harper

This is part of a se­ries. Go to the start.

Expression rules (or more com­monly, ExprLinters) are Harper rules that use de­clar­a­tive ex­pres­sions to find and fix gram­mat­i­cal er­rors. They’re halfway be­tween a “phrase cor­rec­tion” and man­u­ally im­ple­ment­ing Linter.

Make sure you prop­erly set up your en­vi­ron­ment.

Before we get started, let’s take a look at the ExprLinter trait. Here’s what it looks like at the time of writ­ing this post.

/// A trait that searches for tokens that fulfil [`Expr`]s in a [`Document`].
///
/// Makes use of [`TokenStringExt::iter_chunks`] to avoid matching across sentence or clause
/// boundaries.
#[blanket(derive(Box))]
pub trait ExprLinter: LSend {
    /// A simple getter for the expression you want Harper to search for.
    fn expr(&self) -> &dyn Expr;
    /// If any portions of a [`Document`] match [`Self::expr`], they are passed through [`ExprLinter::match_to_lint`] to be
    /// transformed into a [`Lint`] for editor consumption.
    ///
    /// This function may return `None` to elect _not_ to produce a lint.
    fn match_to_lint(&self, matched_tokens: &[Token], source: &[char]) -> Option<Lint>;
    /// A user-facing description of what kinds of grammatical errors this rule looks for.
    /// It is usually shown in settings menus.
    fn description(&self) -> &str;
}

The struc­ture of the trait re­veals some of the be­hind-the-scenes work Harper is do­ing for you. There are three phases:

1. You pro­vide Harper an Expr. It will it­er­ate through the doc­u­ment, look­ing for to­ken se­quences that match your ex­pres­sion.
2. Any and all matches are passed to match_to_lint. From there, you can per­form op­tional ad­di­tional val­i­da­tion to con­firm that the to­kens re­ally do rep­re­sent a gram­mat­i­cal er­ror. If so, re­turn None. Otherwise, re­turn a Lint with any sug­ges­tions that may fix the prob­lem.
3. Harper will han­dle every­thing else. It will show UI, re­for­mat text, and set­tings menus to the user. It will also per­form ag­gres­sive caching on the first two steps, so any mod­i­fi­ca­tions to the doc­u­ment have a neg­li­gi­ble per­for­mance im­pact.

Let’s Get Started

Now that we’ve re­viewed the es­sen­tials, let’s im­ple­ment an ExprLinter.

Before we can write a sin­gle line of code, we need a gram­mat­i­cal rule of in­ter­est. I’m go­ing to pay a visit to the Harper is­sue board.

After look­ing through a few op­tions, I think #1513 is a good can­di­date. We are look­ing for miss­ing prepo­si­tions be­tween an ad­jec­tive and a sub­ject.

To get started, we’ll cre­ate a file un­der harper-core/src/linting called missing_preposition.rs and add it to the par­ent Rust mod­ule. I’ll paste the tem­plate into the file:

pub struct MissingPreposition {
    expr: Box<dyn Expr>,
}

impl Default for MissingPreposition {
    fn default() -> Self {
        let expr = todo!();

        Self {
            expr: Box::new(expr),
        }
    }
}


impl ExprLinter for MissingPreposition {
    fn expr(&self) -> &dyn Expr {
        self.expr.as_ref()
    }

    fn match_to_lint(&self, matched_tokens: &[Token], _source: &[char]) -> Option<Lint> {
        unimplemented!()
    }

    fn description(&self) -> &'static str {
        unimplemented!()
    }
}

I like to start by build­ing out a few test cases be­fore work­ing on the ac­tual code. We get some for free from the GitHub is­sue:

#[test]
fn fixes_issue_1513() {
    assert_lint_count(
        "The city is famous its beaches.",
        MissingPreposition::default(),
        1,
    );
    assert_lint_count(
        "The students are interested learning.",
        MissingPreposition::default(),
        1,
    );
}

Obviously, these tests will fail if we try to run cargo test, but at this point you should do so any­way to make sure your tool­chain is work­ing.

Writing our Expression

The heart of this gram­mat­i­cal rule is the Expr (pronounced ex­pres­sion). There are a num­ber of ways to go about mak­ing one of these. The sim­plest (and most com­mon by far) is to put to­gether a SequenceExpr.

In our case, we’re look­ing for miss­ing prepo­si­tions be­tween an ad­jec­tive and a noun. A good ex­pres­sion to start with could look like:

impl Default for MissingPreposition {
    fn default() -> Self {
        let expr = SequenceExpr::default()
            .then(UPOSSet::new(&[UPOS::ADJ]))
            .t_ws()
            .then(UPOSSet::new(&[UPOS::NOUN, UPOS::PRON, UPOS::PROPN]));

        Self {
            expr: Box::new(expr),
        }
    }
}

We’re us­ing a UPOSSet here, which is an­other kind of Expr that looks for spe­cific parts of speech. The name de­rives from the Universal Dependencies tag sys­tem. Any to­kens tagged with any of the op­tions we’ve pro­vided to the UPOSSet will match.

However, it’s easy to cre­ate an ex­am­ple that this ex­pres­sion matches against, but does­n’t con­tain a gram­mat­i­cal er­ror. We call this a false pos­i­tive. Let’s write one and add it to our test suite.

#[test]
fn allows_terrible_stuff() {
    assert_no_lints(
        "Either it was terrible stuff or the whiskey distorted things.",
        MissingPreposition::default(),
    );
}

From here, you should use your brain to con­tin­u­ously re­fine the ex­pres­sion into some­thing that main­tains a low false-pos­i­tive rate while re­main­ing use­ful. Here’s what I set­tled on:

impl Default for MissingPreposition {
    fn default() -> Self {
        let expr = SequenceExpr::default()
            .then(
                AnchorStart.or(SequenceExpr::default()
                    .then(UPOSSet::new(&[UPOS::DET]))
                    .t_ws()),
            )
            .then(UPOSSet::new(&[UPOS::NOUN, UPOS::PRON, UPOS::PROPN]))
            .t_ws()
            .then(UPOSSet::new(&[UPOS::AUX]))
            .t_ws()
            .then(UPOSSet::new(&[UPOS::ADJ]))
            .t_ws()
            .then(UPOSSet::new(&[UPOS::NOUN, UPOS::PRON, UPOS::PROPN]))
            .then_optional(AnyPattern)
            .then_optional(AnyPattern);

        Self {
            expr: Box::new(expr),
        }
    }
}

Now that we have an ef­fec­tive ex­pres­sion as a base, let’s fill out the re­main­ing fields. I found check­ing for an ad­po­si­tion re­duced the false-pos­i­tive rate, and it was eas­i­est to add it to the match_to_lint func­tion.

impl ExprLinter for MissingPreposition {
    fn expr(&self) -> &dyn Expr {
        self.expr.as_ref()
    }

    fn match_to_lint(&self, matched_tokens: &[Token], _source: &[char]) -> Option<Lint> {
        if matched_tokens.last()?.kind.is_upos(UPOS::ADP) {
            return None;
        }

        Some({
            Lint {
                span: matched_tokens[2..4].span()?,
                lint_kind: LintKind::Miscellaneous,
                suggestions: vec![],
                message: "You may be missing a preposition here.".to_owned(),
                priority: 31,
            }
        })
    }

    fn description(&self) -> &'static str {
        "Locates potentially missing prepositions."
    }
}

That’s it! We’ve writ­ten our rule.

Don’t for­get to reg­is­ter your rule and add some more tests be­fore open­ing PR. Make sure you take a look at the pull re­quest to see the fin­ished rule.