go-bungo-tflite

Guess which 文豪 wrote a Japanese sentence with a neural network, in Go only.

A sibling project of go-fizzbuzz-tflite, go-kani-tflite, go-iris-tflite and go-sazae-tflite: the model is trained in pure Go with a hand-written backprop/Adam implementation — no Python and no TensorFlow — and the .tflite flatbuffer is generated directly from Go, then run with go-tflite.

1. cmd/fetch downloads public-domain texts from Aozora Bunko (青空文庫) through the official index CSV — by default 40 works each by 太宰治, 芥川竜之介, 夏目漱石, 宮沢賢治, 江戸川乱歩, 夢野久作, 坂口安吾, 泉鏡花, 岡本綺堂 and 海野十三 — strips ruby and annotations, splits them into sentences, and writes data/bungo.tsv (~250k sentences). Only 新字新仮名 texts are used, so the model cannot cheat by classifying the orthography instead of the style. Any authors can be substituted with -authors.
2. There is no tokenizer and no dictionary: a sentence becomes 1024 numbers — character bigrams hashed into 1020 buckets plus four surface statistics (length, hiragana/katakana/punctuation ratios).
3. cmd/train trains dense(1024→64) tanh, dense(64→10) softmax — 66k parameters. The train/test split is by work, never by sentence, so the model is graded on books it has never read.
4. The command classifies its arguments, or every line piped to it, with go-tflite, using the label file written next to the model.

One sentence from a book the model never saw, ten authors, chance 10%:

test accuracy: 20376/39502 (51.6%)
               0     1     2     3     4     5     6     7     8     9
0 太宰治　　  47.9   3.5   2.1   3.9  10.6   2.4  13.5   7.6   3.1   5.5
1 芥川竜之介   6.3  28.6  14.8   5.3   3.8   8.9   6.4   9.6   6.0  10.5
2 夏目漱石　   5.0   8.6  31.4   5.8   2.9  10.7  11.0   9.0   9.3   6.3
3 宮沢賢治　   5.0   3.1   5.9  51.0  13.4   4.7   3.3   4.3   5.8   3.4
4 江戸川乱歩   5.7   2.5   1.1   6.4  68.4   2.1   4.8   3.4   2.0   3.6
5 夢野久作　   2.8   4.9  10.9   7.5   3.0  44.4   6.5   4.2   6.7   9.1
6 坂口安吾　   8.7   4.7  10.1   3.5   4.9   9.8  35.1   5.2   6.5  11.5
7 泉鏡花　　   8.7   5.9   3.6   3.2   5.7   6.0   5.9  52.7   3.2   4.9
8 岡本綺堂　   3.0   4.7  15.8   4.6   2.3   4.7   6.5   3.1  48.5   6.8
9 海野十三　   8.1   6.4   7.1   4.8   7.7   9.2  12.6   5.2   6.1  32.7

The confusion matrix doubles as literary criticism: the author 芥川 is mistaken for most often is 漱石 — whose disciple he was.

Requires libtensorflowlite_c.so (see go-tflite for how to build it).

$ go run ./cmd/fetch     # download from Aozora Bunko (cached, not distributed here)
$ go run ./cmd/train     # optional: bungo_model.tflite is checked in
$ go run . 木曾路はすべて山の中である。
  夏目漱石　  51.3%
  坂口安吾　  18.4%
  泉鏡花　　  12.4%
  芥川竜之介   7.9%
  海野十三　   5.8%

The top five candidates are listed — the model has no idea that 島崎藤村 exists, so ranked guesses are more honest than a single confident answer. -all prints every author. Lines on standard input are classified one by one; when the input comes through a pipe (and is therefore not already on your screen), each sentence is printed above its candidates:

$ echo 何かの文章 | go run . -all

Short famous lines are usually guessed wrong — 「メロスは激怒した。」 comes back as 江戸川乱歩 — because nine characters of bigrams carry almost no style. Give it a full paragraph and it does much better; that gap between one-liner and paragraph is exactly what 51.6% per sentence looks like.

All texts come from Aozora Bunko, transcribed and proofread by its volunteers (see data/ATTRIBUTION.md). The scraped dataset is not redistributed here; run go run ./cmd/fetch once to build it locally (downloads are cached under data/cache). Only the trained model and its label file are checked in.

MIT (the code; the texts are public domain via Aozora Bunko)

Yasuhiro Matsumoto (a.k.a. mattn)