Skip to content

Commit 8198791

Browse files
andrew-polkclaude
andcommitted
chore: BL-15916 separate raw source data files from processed data files
https://issues.bloomlibrary.org/youtrack/issue/BL-15916 The card notes that the raw source files and the files our pipeline generates from them lived together in one folder and should be separated. Move the raw source files into a new language-data/source/ subfolder, leaving the generated files (languageData.json, equivalentTags.json), which the runtime imports, in language-data/: - language-data/source/langtags.json - language-data/source/iso-639-3.tab - language-data/source/iso-639-3-macrolanguages.tab Updated the source-file import/read paths in langtagProcessing.ts and langtagProcessingHelpers.ts, and the Data sources section of the README. Verified: re-running the langtag-processing pipeline from the new paths produces byte-identical languageData.json and equivalentTags.json; tsc typecheck passes; the full find-language test suite passes (174/174). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 9a57b24 commit 8198791

6 files changed

Lines changed: 6 additions & 6 deletions

File tree

components/language-chooser/common/find-language/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -206,7 +206,7 @@ For languages that are representative for a macrolanguage, however, we use the i
206206

207207
## Data sources
208208

209-
We get our data from the [langtags repository](https://github.com/silnrsi/langtags/tree/master) and [ISO 639-3](https://iso639-3.sil.org/). All live in the [language-data](./language-data/) folder. In the future, we plan to automate the update of these data files, but currently it is done manually.
209+
We get our raw source data from the [langtags repository](https://github.com/silnrsi/langtags/tree/master) and [ISO 639-3](https://iso639-3.sil.org/). These raw files live in the [language-data/source](./language-data/source/) folder, kept separate from the processed files (see below) that our pipeline generates from them. In the future, we plan to automate the update of these data files, but currently it is done manually.
210210

211211
- langtags.json
212212
- source: https://ldml.api.sil.org/langtags.json
@@ -218,7 +218,7 @@ We get our data from the [langtags repository](https://github.com/silnrsi/langta
218218
- source: https://iso639-3.sil.org/sites/iso639-3/files/downloads/iso-639-3-macrolanguages.tab
219219
- documentation: https://iso639-3.sil.org/code_tables/download_tables
220220

221-
Currently also in the folder, data files compiled by our language data processing (see section above):
221+
The processed data files compiled by our language data processing (see section above) live in the [language-data](./language-data/) folder:
222222

223223
- languageData.json
224224
- equivalentTags.json

components/language-chooser/common/find-language/language-data/iso-639-3-macrolanguages.tab renamed to components/language-chooser/common/find-language/language-data/source/iso-639-3-macrolanguages.tab

File renamed without changes.

components/language-chooser/common/find-language/language-data/iso-639-3.tab renamed to components/language-chooser/common/find-language/language-data/source/iso-639-3.tab

File renamed without changes.

components/language-chooser/common/find-language/language-data/langtags.json renamed to components/language-chooser/common/find-language/language-data/source/langtags.json

File renamed without changes.

components/language-chooser/common/find-language/scripts/langtagProcessing.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ import {
1818
} from "./langtagProcessingHelpers";
1919

2020
import fs from "fs";
21-
import langTagsJson from "../language-data/langtags.json" with { type: "json" };
21+
import langTagsJson from "../language-data/source/langtags.json" with { type: "json" };
2222

2323
function bestAutonymFromEntry(entry: any) {
2424
return entry.localnames ? entry.localnames[0] : entry.localname;

components/language-chooser/common/find-language/scripts/langtagProcessingHelpers.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { iso15924 } from "iso-15924";
2-
import langTagsJson from "../language-data/langtags.json" with { type: "json" };
2+
import langTagsJson from "../language-data/source/langtags.json" with { type: "json" };
33
import fs from "fs";
44
import { ILanguage, IScript, LanguageType } from "../findLanguageInterfaces";
55
import {
@@ -78,7 +78,7 @@ export const scriptNames: { [key: string]: string } = iso15924.reduce(
7878
-- E(xtinct), H(istorical), L(iving), S(pecial)
7979
*/
8080
const isoCodesDetailsFile = fs.readFileSync(
81-
"language-data/iso-639-3.tab",
81+
"language-data/source/iso-639-3.tab",
8282
"utf8"
8383
);
8484

@@ -140,7 +140,7 @@ for (const line of isoCodesDetailsFile.split("\n")) {
140140
}
141141

142142
const macrolangMappingFile = fs.readFileSync(
143-
"language-data/iso-639-3-macrolanguages.tab",
143+
"language-data/source/iso-639-3-macrolanguages.tab",
144144
"utf8"
145145
);
146146
export const indivlangsToMacrolangs: { [key: string]: string } = {};

0 commit comments

Comments
 (0)