Skip to content

Rank the suggester's common given names from ORCID, and drop the last nam_dict data - #1374

Open
derek73 wants to merge 17 commits into
masterfrom
feat/orcid-common-given-names
Open

derek73 wants to merge 17 commits into
masterfrom
feat/orcid-common-given-names

Conversation

@derek73

@derek73 derek73 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Based on master since #1371 merged; master is merged in, and the diff is this PR's own.

What

The group suggester's common given names (_COMMON_GIVEN in nameparser/config/_order_names.py) are now ranked from ORCID's public data file (CC0), not from nam_dict's country columns (GNU FDL). After this PR, _order_names.py holds no nam_dict data.

tools/vocab/order_names.py builds the table like this:

  • For each country an ORCID holder states (250 in the local corpus), it ranks the words typed first in the given-name box.
  • Only the first word is ranked. A ranking of every word let in East Slavic patronymics (викторовна).
  • A word must be typed that way at least 3 times in the country to count, and each country contributes its top 500.
  • Each word is counted under the spelling the suggester looks it up by: _suggest._EDGES stripped, then full stops, then folded. A test holds the two spellings together.

The TABLES filter is unchanged: at most a 5% family-box share over at least 20 uses, two letters or more, and no listed surname. ORCID needs two more refusals:

  • a word holding a full stop (dr.ahmed, م.م);
  • a word the default lexicon reads as a title outside titles_ambiguous (prof, dra, lic, engr, assoc, asst, lady). Holders type these before the name, and in the first build Prof. Li Jing cast a given-first vote.

The table holds 6,409 names, where nam_dict's held 4,262: 4,235 shared, 2,174 new and 27 left. The new names are romanized Chinese and Korean, Indian, African and Central Asian given names, plus Cyrillic spellings. The table is the project's own measurement over a CC0 file, so it carries no licence of its own (decisions/shipped-data-licences, FACTS WE MEASURE).

Licences

#1372 (merged) dropped nam_dict's familiar pairs, so with this PR no nam_dict data is left in the package. The merge commit b6816da therefore removes the licence material:

  • LICENSE-nam_dict;
  • README's bullet;
  • GFDL-1.2-no-invariants-or-later from license, and the file from license-files;
  • its _THIRD_PARTY row in tests/v2/test_licensing.py.

The expression is now LGPL-2.1-or-later AND Apache-2.0 AND CC-BY-4.0 AND OGDL-Taiwan-1.0, and no release ever named the GFDL. The release-log bullet from #1370 says so.

Moves

uv run python tools/differential/suggest_measure.py --base <checkout of 5a713720>, run against #1371's tree, whose tables differ only in _COMMON_GIVEN:

Order table. 159 cells moved: 147 up and 12 down. The largest fall is 0.6 points.

  • Same-order rows. Four same-order cells fell, each by one group in 300. The worst same-order row outside the three anti-goal rows still reads 98%.
  • PatentsView's Indian inventors are first read right in 95% of groups at 50 names, where it took 200. PSC's Indian filers still take 200, and no other population's first size moved.
  • Largest gains. These are where nam_dict's columns were thin. PSC's Indian filers written given-first under a family-first base went from 30.0 / 59.7 / 93.3% to 45.7 / 84.0 / 99.3% at 10 / 20 / 50 names. PSC's Nigerian filers written family-first went from 35.3% to 62.7% at 10 names. PatentsView's Indian inventors went from 52.3% to 77.7% at 20.

Per name. The table cast 48,202 right and 1,087 wrong given-name votes, against 45,079 and 973 before (2.21% wrong, from 2.11%).

  • Within a population, the wrong share held or fell in most rows.
  • The names read wrong rose most for PSC's Nigerian filers (16→25 and 19→26 of 3,000) and for PatentsView's Indian inventors written family-first (42→52).

Parliaments. The order counts of 12 lists moved, and one suggestion changed. Mongolia's Khural is now decided family-first by two names that end in a listed given name, which is how locales.MN reads it (#1237); two is the order rule's minimum. The Duma casts 383 given-first votes, and its suggestion, decided by the patronymic rule, does not move.

Anti-goal rows. The rows of ORCID holders who typed their given name in capitals rose: from 26.7% to 30.0% at 2 names. Parser.suggest's docstring now quotes 30%.

Caveat. The orcid-* populations are drawn from the same ORCID sample, so their rows are not an independent test of this table. The anti-goal populations (PatentsView, PSC, the Czech candidates, the Vietnamese names) are.

Docs

  • decisions/group-suggestion:
    • a 2026-11-12 GIVEN NAMES FROM ORCID entry, marked proposed, for review;
    • two Standing lines;
    • a head note on the TABLES entry.
  • decisions/shipped-data-licences: an entry, and a head note on Name the licences of the shipped data in the package metadata #1370's table.
  • Parser.suggest's docstring and docs/suggest.rst: the Indian figures, re-taken.

Checks

Reviews

  • design-docs-reviewer. Its findings were all fixed:
  • pr-review-toolkit code-reviewer. It found that the tool's spelling differed from the suggester's (ala' was unreachable), the docstrings missed the title refusal, listed_titles' docstring did not match its code, and the floor was stale. All fixed.
  • A code-reviewer pass on the fix commit. It found that mst and ace entered the table, and since a listed name is never read past as a post-nominal, Mary Smith MST casts no vote. This is recorded as an accepted cost beside asa and cem. It also found that the spelling test checked a copy of the tool's formula; it now checks every row against _suggest._name_words.
  • A review of the merge that removes the nam_dict licence. It found the release-log sentence calling the familiar forms rebuilt where they were dropped; fixed.

🤖 Generated with Claude Code

derek73 and others added 15 commits November 12, 2026 20:47
The licence expression read LGPL-2.1-or-later alone while the wheel
ships data under five other licences. It now reads
LGPL-2.1-or-later AND Apache-2.0 AND GFDL-1.2-no-invariants-or-later
AND CC-BY-4.0 AND OGDL-Taiwan-1.0 AND Unicode-3.0.

- LICENSE-nam_dict: nam_dict's notices and the GFDL 1.2 text, which
  its terms ask to travel with the familiar forms and the suggester's
  common given names and which no file in the tree held.
- LICENSE-unicode: the Unicode License v3 notice for the Unihan data,
  in locales/zh.py's simplified surname forms since 2.5.0 and in the
  suggester's pinyin surnames; zh.py's docstring now credits it.
- README.rst's License section lists every source.
- tests/v2/test_licensing.py holds the licence files, license-files,
  the expression, the README and the crediting modules together.
- decisions/shipped-data-licences records the sources, what each
  licence asks and how it is met; AGENTS.md says what a new source
  needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Data the project derives by measuring is its own, its sources credited
as a courtesy; a curated list copied whole keeps its licence. Under
that rule nam_dict's familiar-form pairs are dropped before 3.1 and
rebuilt from CC0 data reviewed by hand.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…in surnames

locales.ZH's simplified surname forms now come from the simplified-
Chinese labels of Wikidata's family-name items, and the group
suggester's pinyin surnames from those items' Hanyu Pinyin statements,
both CC0, so the package ships no Unihan-derived data and Unicode-3.0
and LICENSE-unicode leave its licence metadata.

Seven simplified forms no Wikidata item attests (买厐盖篮芈诸谌) leave
locales.ZH; over 322,895 corpus names holding a Han character, 9 move,
all foreign names the pack had cut after 盖. The pinyin block holds
the surname readings (曾 zeng, 单 shan) where Unihan's first reading
gave ceng and dan; suggest_measure moved 37 cells, no quoted figure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the review's doc findings

Both tools reused a cached answer whenever it covered the forms asked,
whatever the query; each now reuses only the answer to the exact
queries it writes, asks in batches of 100 (the whole set outran the
service's time limit), and refuses an answer holding a row that is not
one, which is how the service's error text reads when it cuts a CSV
answer short. A form an item writes the same in both scripts now
counts as its own simplified form, as Unihan's table listed it, so it
leaves the form two candidates; the shipped blocks do not move.

The group-suggestion entry counted listing lines, not cells (59 moved,
54 up, 5 down), left out `an` and named two anti-goal rows of three;
the #695 and TABLES entries get head notes, AGENTS.md says Unihan has
left, and the release-log bullet says what 2.5.0-3.0.1 declared.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… batch be empty

The order_names tests patched tool.name_usage, but ask() calls the
name_usage zh_surnames holds, which another test module's loader can
make a different object, sending the tests to Wikidata; they now patch
that copy and refuse the network. The pinyin cache test now asks a
subset of what it cached, the case the parent answered from the cache,
and a changed pinyin query. A batch of 100 items may hold none with a
reading, so an empty pinyin batch is an answer. W1's Open line records
that #981's mainland ranking would restore the seven dropped forms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not nam_dict

_COMMON_GIVEN now ranks, per country an ORCID holder states, the words
typed first in the given-name box (CC0), with the same family-share
filter, a minimum of 3 uses in the country, and two refusals the new
source needs: a word holding a full stop, and a word the default
lexicon reads only as a title (Prof., Dra., Lic. typed before the
name). The table holds 6,407 names where nam_dict's held 4,262.

suggest_measure: 158 cells moved, 146 up and 12 down (at most 0.6
points); PatentsView's Indian inventors are read right in 95% of groups
from 50 names where it took 200, and Indian and Nigerian groups gain up
to 25 points at 10 and 20 names. The worst same-order row outside the
anti-goal rows still reads 98%.

LICENSE-nam_dict now names the familiar forms alone; GFDL stays in the
expression for them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…with-wikidata

# Conflicts:
#	AGENTS.md
#	README.rst
#	docs/design/decisions/shipped-data-licences.md
#	docs/release_log.rst
#	nameparser/locales/zh.py
#	pyproject.toml
#	tests/v2/test_licensing.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…w's figures

The tool stripped full stops and commas from a word, where the
suggester strips quotes and brackets first, so `ala'` entered a table
the suggester could never match it in; `spelling()` now keys a word as
the suggester reads it, with a test holding the two together. The
table holds 6,409 names; the floor rises to 5,000, a third of a cut
input. listed_titles' docstring says what it computes, and the filter
docstrings name the title refusal. The given-name Standing line is
proposed, for review; the entry counts 159 cells (147 up) and 12
parliament lists, records the anti-goal rows that rose, `mst`, and the
#1370 entry gets a head note; Parser.suggest quotes the re-taken 30%.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…with-wikidata

# Conflicts:
#	README.rst
#	docs/design/decisions/shipped-data-licences.md
…d ace as an accepted cost

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…' into feat/orcid-common-given-names

# Conflicts:
#	LICENSE-nam_dict
#	README.rst
#	docs/design/decisions/group-suggestion.md
#	docs/design/decisions/shipped-data-licences.md
… file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added vocabulary Fixed by a word-list entry: per-word vocabulary and language packs suggester The group suggester: Parser.suggest() and its signals labels Nov 13, 2026
@derek73 derek73 self-assigned this Nov 13, 2026
Base automatically changed from fix/replace-unihan-with-wikidata to master November 13, 2026 06:36
derek73 and others added 2 commits November 12, 2026 22:37
…iven-names

# Conflicts:
#	AGENTS.md
#	README.rst
#	docs/design/decisions/group-suggestion.md
#	docs/design/decisions/shipped-data-licences.md
#	docs/release_log.rst
#	pyproject.toml
#	tests/v2/test_licensing.py
#	tests/v2/test_order_names_tool.py
#	tools/vocab/order_names.py
The ORCID table lists `xiaoming`, so `print(suggestion)` in
getting-started.rst and suggest.rst counts two given-name votes for
family first, where the expected output said one; the Sphinx doctest
step failed in every build job. AGENTS.md now says a suggester table
change needs that step run too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

suggester The group suggester: Parser.suggest() and its signals vocabulary Fixed by a word-list entry: per-word vocabulary and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant