Language Tech
Homebrewed Nihongo: A DIY Guide to Japanese Language Tech
What it actually takes to parse, look up, and render Japanese text in the browser, without a backend.
By My Senpai Team · Published May 28, 2026 · Updated Jun 2, 2026 · 14 min read
A hands-on guide to the four technical layers every Japanese reading or learning tool has to solve:
word segmentation, dictionary lookup, furigana rendering, and OCR. Covers
Kuromoji,
JMdict,
sql.js, and
Tesseract.js.
A live furigana demo is embedded in Section 4.
Maybe it's the way Japanese learning apps seem to prioritize cloud tracking and monthly fees over the content itself. Maybe you want a reading tool tuned to your specific needs and the content you actually like to read. Not the content a subscription model needs you to consume.
You've started to think that building your own tool is the next logical step. That's why you're here. Here's what you need to know to get started.
This article covers four technical problems you will run into when building any Japanese reading or learning tool: how to split Japanese text into words, how to look those words up without a server, how to generate furigana that is actually correct, and how to extract text from images. It does not cover which existing tool you should use instead, or how to study Japanese more effectively. Those articles exist. This one assumes you've already made the decision.
The Segmentation Problem
Take the string 日本語を勉強するのは難しい. Find the words in it. There are no spaces, no capital letters, no punctuation to use as reference points. If you grew up reading English, your brain is looking for gaps that are not there.
Hand-counting characters works until it doesn't. Compound words span multiple kanji. Verb conjugations attach directly to their stems. Particles sit flush against the words before them with no visible seam. You cannot split on anything you recognize. This is the segmentation problem. It is the first thing your app needs to solve before anything else: dictionary lookup, furigana generation, vocabulary flagging. All of it depends on having clean word boundaries first.
NLP (Natural Language Processing)
A field of computer science focused on enabling programs to understand and work with human language. For Japanese tools, the relevant subfield is morphological analysis: breaking text into its smallest meaningful units (morphemes) and labeling each one with its grammatical role.
Tokenization and Morphological Analysis
Tokenization is the process of splitting a string of text into individual tokens (words, particles, punctuation). Morphological analysis goes further: it also identifies each token's dictionary form, reading, and part of speech. For Japanese, these two tasks happen together because word boundaries cannot be determined without grammar knowledge.
IPAdic
IPAdic is a morphological dictionary compiled by the Nara Institute of Science and Technology. It contains roughly 400,000 entries covering the surface forms, readings, and part-of-speech data that
Kuromoji uses to identify word boundaries. IPAdic was last updated in 2007, which means it has no entries for vocabulary coined after that date.
Kuromoji is a JavaScript port of the original Java library, published in 2014 and available on npm. It uses IPAdic to perform morphological analysis. Given a string, it returns an array of token objects, one per morpheme. The token for 勉強 looks like this:
{
surface_form: "勉強", // text as written
reading: "ベンキョウ", // katakana pronunciation
basic_form: "勉強", // dictionary form, used for lookups
pos: "名詞" // part of speech
}
Install Kuromoji:
npm install kuromoji
import kuromoji from 'kuromoji';
kuromoji.builder({ dicPath: 'node_modules/kuromoji/dict' }).build((err, tokenizer) => {
const tokens = tokenizer.tokenize('日本語を勉強するのは難しい');
tokens.forEach(t => console.log(t.surface_form, t.reading, t.pos));
// 日本語 ニホンゴ 名詞
// を ヲ 助詞
// 勉強 ベンキョウ 名詞
// する スル 動詞
// 難しい ムズカシイ 形容詞
});
Two things that will trip you up
First: Kuromoji is synchronous and slow on first call. The dictionary files are roughly 5 MB. If you initialize the tokenizer on the main thread, the UI freezes during load. Put it in a Web Worker.
Second: IPAdic has wrong readings for some date and time expressions. 一日 should not read いちにち. 四時 should not read しじ. These require a manual correction table. YoMoo ships around 40 of these rules. You will find yours as users encounter them.
The Dictionary Dilemma
You have tokens. Each one has a surface_form and a basic_form. Now you need meanings. The obvious choice is the Jisho.org API: free, returns JSON, requires no authentication. It also has no SLA, no documented rate limits, and goes down periodically. Every network call is a failure point. If your tool needs to work offline, the Jisho API disqualifies itself on that criterion alone.
The alternative is to ship the dictionary yourself. JMdict is roughly 190,000 entries, available as a free download under Creative Commons, and already powering every serious Japanese learning tool you have heard of. Compiled into SQLite, it is about 20 MB. With sql.js, you can query it entirely in the browser.
WebAssembly (WASM)
WebAssembly is a binary instruction format that runs in the browser at near-native speed. It lets you compile code originally written in C or C++ and run it in a web page. For Japanese tools, this matters because the libraries that do the heavy lifting (SQLite among them) are native code. WASM is how they get into the browser.
SQLite-in-WASM (sql.js)
sql.js is a version of the SQLite database engine compiled to WebAssembly. It loads a
.db file into memory and lets you run SQL queries against it entirely in the browser, with no server. For dictionary lookup, this means you ship a 20 MB file once and query it instantly on every subsequent word tap.
JMdict
JMdict is a community-maintained Japanese-English dictionary with over 190,000 entries. It is the dataset behind
Jisho,
Yomitan, and most serious Japanese learning tools. The project is maintained by the Electronic Dictionary Research and Development Group and licensed under Creative Commons Attribution-ShareAlike.
Install sql.js:
npm install sql.js
import initSqlJs from 'sql.js';
const SQL = await initSqlJs({ locateFile: f => `/wasm/${f}` });
const buf = await fetch('/dict/jmdict.db').then(r => r.arrayBuffer());
const db = new SQL.Database(new Uint8Array(buf));
function lookupWord(basicForm, reading) {
const r1 = db.exec(
'SELECT kanji_reading, reading, glosses, pos, jlpt FROM jmdict WHERE kanji_reading = ?',
[basicForm]
);
if (r1[0]?.values.length) return r1[0].values[0];
const r2 = db.exec(
'SELECT kanji_reading, reading, glosses, pos, jlpt FROM jmdict WHERE reading = ?',
[reading]
);
return r2[0]?.values[0] ?? null;
}
The kana collision problem
Pass 2, the kana fallback, creates a subtle bug class. 起きて has the reading おきて, which also matches 掟 (meaning "law" or "rule"). If your app returns 掟 when the user tapped 起きて, you have a false positive from the kana fallback. The fix is a span-scoring layer on top of the SQL query that penalizes kana-only matches when a kanji match is available elsewhere in the token list.
The 20 MB download is the main cost of this approach. On a first visit, users wait. On every subsequent lookup, they pay nothing. For an immersion tool aimed at learners reading native content daily, that tradeoff is almost always acceptable.
The Furigana Nightmare
You have tokens with readings. The natural next step is to wrap every kanji sequence in a <ruby> tag. For a simple word, it looks correct:
<ruby>日本語<rt>にほんご</rt></ruby>
It breaks on compound verbs. Kuromoji returns 起きておいた as four separate tokens: 起き, て, おい, た. If you wrap each token independently, you fragment the compound verb and misalign the okurigana. The verb becomes unreadable. Any reader who knows Japanese will see the furigana is wrong, even without being able to articulate why.
The pipeline
A working furigana generator needs at least four steps:
function generateFurigana(text, tokenizer, db) {
const tokens = tokenizer.tokenize(text);
// Step 1: patch known irregular readings
const corrected = applyCorrections(tokens);
// Step 2: collapse verb + auxiliary chains into single tokens
const merged = mergeVerbAuxiliaryChains(corrected);
// Step 3: score candidate spans against JMdict
const groups = findBestTokenGroups(merged, db);
// Step 4: assemble ruby tags from the winning spans
return groups.map(g => toRubyHTML(g)).join('');
}
Step 2 tracks a specific set of auxiliary verbs that combine with the preceding verb: いる, ある, おく, しまう, みる, くる, もらう, あげる, やる. Each combination collapses into a single token before the span-scorer runs. Without this step, compound verbs produce fragmented furigana on every occurrence.
Step 3 scores spans of one to six tokens. Longer compound matches score higher because JMdict compound entries are more specific than single-word fallbacks. This is what lets 起きておく match as one entry rather than three.
Scale note: This pipeline works well for texts under roughly 50,000 characters. Above that threshold, the span-scorer runs against the full token list on each call and becomes the bottleneck. If you are processing long texts, chunk the input into paragraphs before passing each one through the pipeline.
Live Demo: Furigana Reader
The interactive furigana reader embedded in the JavaScript version of this page runs the same Kuromoji and Jisho.org stack described in Sections 1 and 2. Paste any Japanese text, tap Analyze, and tap any word for an English definition. This is roughly what the tokenization and lookup layer looks like when connected to a minimal UI.
The demo calls the Jisho.org API and requires a network connection. YoMoo replaces this lookup with a local sql.js query against jmdict.db for offline support. The tokenization layer (Kuromoji) is identical in both cases.
OCR for Immersion
You want to read a manga panel. The text is inside an image. You cannot select it, copy it, or pass it through a tokenizer. You need to extract it first.
Optical character recognition for Japanese is harder than for Latin scripts. The character set is larger, fonts vary widely, and speech bubble text is often small, curved, or printed over artwork. The tools exist. The accuracy ceiling is lower than you might expect.
OCR (Optical Character Recognition)
The process of extracting text from images. OCR engines are trained on image-text pairs and learn to recognize character shapes. For Japanese, a capable engine must recognize several thousand kanji in addition to hiragana and katakana. Accuracy depends heavily on font size, image contrast, and whether the training data covered the specific style of text you are working with.
Tesseract.js is a WebAssembly port of the Tesseract OCR engine with a trained Japanese language model. Install it:
npm install tesseract.js
import { createWorker } from 'tesseract.js';
async function extractJapanese(imageSource) {
const worker = await createWorker('jpn');
const { data } = await worker.recognize(imageSource);
await worker.terminate();
return data.text;
}
Pre-processing the image before OCR improves results on low-quality scans:
function preprocessForOCR(canvas) {
const scaled = document.createElement('canvas');
scaled.width = canvas.width * 2;
scaled.height = canvas.height * 2;
scaled.getContext('2d').drawImage(canvas, 0, 0, scaled.width, scaled.height);
const ctx = scaled.getContext('2d');
const imgData = ctx.getImageData(0, 0, scaled.width, scaled.height);
for (let i = 0; i < imgData.data.length; i += 4) {
const g = 0.299 * imgData.data[i] + 0.587 * imgData.data[i+1] + 0.114 * imgData.data[i+2];
imgData.data[i] = imgData.data[i+1] = imgData.data[i+2] = g;
}
ctx.putImageData(imgData, 0, 0);
applyAdaptiveThreshold(scaled, 25, -10);
return scaled;
}
Accuracy warning: Tesseract.js accuracy on manga is mediocre. Small fonts, speech bubble borders, and stylized handwriting all degrade output significantly. For clean printed text with high contrast, results are acceptable. For everything else, expect to correct output manually. The pre-processing steps above help. They do not close the gap with native mobile OCR.
Android alternative: Google ML Kit's Japanese text recognition is substantially more accurate than Tesseract.js and handles handwriting. If you are building a native Android app, use ML Kit instead. The trade-off is that it ties your implementation to Android. Tesseract.js is the right choice for a browser-based tool.
The Asset Footprint
You are building a client-side tool. Every kilobyte matters, especially if you are already asking the user to download a 20 MB dictionary and a 5 MB morphological analyzer on their first visit. You will likely use SVGs for your UI icons, buttons, and diagrams.
Raw SVGs exported from design tools like Figma or Illustrator are bloated. They contain editor metadata, empty tags, hidden layers, and unoptimized paths. If you inline them directly into your React components, your bundle size inflates for no structural reason.
Before you commit those assets, you need to strip the garbage out. Run your vectors through an aggressive minifier like the Icosix SVG Optimizer. It strips out the XML cruft and mathematically simplifies the paths without destroying the visual integrity of the image. Keep your bundle tight. Your users are already downloading enough.
Frequently Asked Questions
Do I need a backend to query JMdict?
No. sql.js loads the entire database into the browser's memory via WebAssembly. Queries run client-side. The tradeoff is a one-time download of roughly 20 MB. Once the file is cached, every lookup is instant and works offline.
Why does Japanese need tokenization when other languages do not?
Japanese is written without spaces between words. The string 日本語を勉強する contains four distinct morphemes but zero visual delimiters. Without a tokenizer, you cannot identify word boundaries reliably, and every downstream task (dictionary lookup, furigana generation, vocabulary flagging) breaks.
What is Kuromoji's biggest limitation?
IPAdic was compiled in 2007. It does not include modern internet vocabulary, recent loan words, or proper nouns coined after that date. Words absent from the dictionary are returned as single-character tokens with no reading. You will need a correction table for common gaps in your content.
Can Tesseract.js read manga panels reliably?
For clean, high-contrast printed text, accuracy is acceptable. For stylized fonts, small speech bubble text, or handwriting, it fails frequently. The pre-processing pipeline helps. It does not close the gap with native ML Kit on Android.
How does YoMoo generate furigana without a third-party library?
It runs a custom pipeline. Kuromoji tokenizes the text. A correction table patches roughly 40 irregular readings. A verb-auxiliary merger collapses multi-token verb chains. A span-scorer picks the best JMdict match per position. Then ruby tags are assembled from the winning span's reading. The full implementation is around 820 lines of vanilla JavaScript.