Tokora

Tokora is a Rust parser-combinator library with on-demand lexing, explicit lookahead and backtracking, configurable diagnostics, and optional Logos and Rowan integrations. Parsers work over a Lexer and Token model, so the same grammar can use a fail-fast runtime emitter or a collecting diagnostic emitter.

Most applications use the maintained Logos adapter:

[dependencies]
tokora = { version = "0.9", features = ["logos"] }

logos is the alias for the logos_0_16 integration, the only Logos major tokora supports. The default std feature remains enabled unless you set default-features = false.

Which version this describes. Every [dependencies] snippet in this README resolves against the latest release, 0.9.1. A feature that main has grown since is named with the version it will arrive in, rather than put in a copyable block you cannot resolve.

• On-demand token flow through InputRef, with explicit cache-backed lookahead and transactions.
• Plain parser functions plus composable sequencing, repetition, delimiters, and deterministic choice.
• Token-level and AST-level Pratt parsing.
• Configurable Fatal, Verbose, Silent, and Ignored diagnostics.
• Recovery, partial-input support, lexer conformance checks, tracing, and a public fuzz harness.
• Optional lossless CST building over the parser's own backtracking (feature rowan): a rewindable event-stream sink where a parser rollback rewinds the half-built tree, and node combinators bracket sub-parses into syntax nodes.
• Optional adapters for Logos, Rowan CSTs, source types, and container types.

A Tokora grammar is ordinary Rust: parser functions and combinators read through InputRef, which pulls tokens from a Lexer on demand and stages tokens in its cache when lookahead or backtracking needs them. peek_then_choice makes a decision from a fixed lookahead window; dispatch_on_kind and fused_dispatch_on_kind route the next token's Token::Kind to exactly one selected branch.

That token-kind dispatch is local to a hand-written combinator grammar. Tokora does not accept an LALR grammar, generate LALR parse tables, or act as an LALR parser generator.

When a grammar needs speculation, it is explicit. attempt and try_attempt commit successful work and roll back a decline or error; Transaction exposes commit and rollback directly. A rollback restores the input position, span, lexer state, token cache, and diagnostics emitted since the checkpoint. Application-owned side effects need their own transaction boundary.

Parsers are generic over their parse context, including the emitter. Parser::new() uses the fail-fast Fatal emitter; Verbose records diagnostics and can continue when the grammar recovers. The same parser functions can therefore serve a runtime parser, compiler front end, or editor integration without a second grammar implementation.

Structured lexer, token, separator, container, and Pratt errors convert into the application's error type through From implementations.

Recovery is explicit: recover restores the failed parse's starting point before running a recovery parser, while inplace_recover continues from the failure position. sync_balanced and skip_then_retry provide nesting-aware synchronization; Verbose records each successful non-empty skipped region once alongside other diagnostics. Incomplete errors are re-raised instead of recovered so unfinished partial input is not discarded.

The Tokora Guide has three parts: ten Calc fundamentals, one anatomy chapter plus four maintained-example walkthroughs (five applied-parser chapters), and an optional Rowan/lossless-CST chapter. The examples below are canonical complete programs; the guide links back to them instead of copying whole files into prose.

The book source lives under tokora/src/guide, and the examples also compile together with cargo test -p tokora --no-default-features --features std,logos,combinators --examples.

The combinator-family gates — combinators and the thirteen families it covers, any through validate — are new in 0.9.0. On 0.8.0 and earlier there are no per-family gates, default is std alone, and every combinator is compiled unconditionally.

Every combinator family is independently gateable so an embedded or no-alloc build compiles only the combinators it calls. What the families sit on stays unconditional: Parser/Parse/parse*, the ParseInput / TryParseInput / ParseChoice traits, and the substrate combinators (expect, delimited, recover, select, opt, padded, node, labelled, …). combinators is a default feature, so a plain dependency line sees the whole surface; a default-features = false build names the families it uses.

Feature aliases select their versioned counterpart; versioned features make the corresponding optional dependency available. tokora::logos and the unversioned tokora::lexer::LogosLexer are available with logos_0_16 and re-export/adapt that version — the only Logos major tokora supports. rowan does not enable logos, and smallvec_1 is the versioned feature that adds alloc.

Tokora's MSRV is Rust 1.95. Tokora's core supports both allocator-free no_std (no_std without alloc) and allocation-enabled no_std (no_std with alloc). Disable default features for allocator-free core use. Enable alloc when a parser, cache, or selected optional facility requires allocation; other optional facilities may require std.

Allocator-free no_std:

[dependencies]
tokora = { version = "0.9", default-features = false }

no_std with alloc:

[dependencies]
tokora = { version = "0.9", default-features = false, features = ["alloc"] }

Neither line enables a combinator family: combinators is a default feature, and both turn the defaults off. Add features = ["combinators"] for the umbrella, or list the families the grammar actually calls, as in features = ["alloc", "many", "map"].

1. Performance - Pull tokens from the lexer on demand and offer fused dispatch where avoiding a peek/cache round trip matters.
2. Predictability - Prefer deterministic lookahead and token-kind dispatch; make speculation explicit and transactional.
3. Composability - Combine small parser functions and combinators; compose focused emitter traits into custom diagnostic strategies.
4. Versatility - Reuse parser functions with fail-fast, collecting, silent, or custom emitters.
5. Flexibility - Work through generic Lexer and Token traits, with optional Logos input and Rowan CST integrations.
6. Correctness - Track spans and structured errors, rewind emitted diagnostics with parser rollbacks, and provide conformance and fuzz test kits.

Tokora takes inspiration from:

• winnow - For ergonomic parser API design
• chumsky - For composable parser combinator patterns
• logos - For high-performance lexing
• rowan - For lossless syntax tree representation

Useful repository checks:

cargo fmt --all --check
cargo test -p tokora --all-features
cargo test -p tokora --no-default-features --features std,logos,combinators --examples
RUSTDOCFLAGS="-D warnings" cargo test -p tokora --all-features --doc
(cd tokora && mdbook build)

The guide is validated both as rustdoc and as an mdBook so API links, local links, chapter order, and Pages output stay aligned.

tokora is under the terms of both the MIT license and the Apache License (Version 2.0).

See the Apache License, Version 2.0 and the MIT license text for details.

Copyright (c) 2026 Al Liu.