Core - unihan_etl.core

Download + ETL UNIHAN into structured format and export it.

unihan_etl.core.not_junk(line)

function

Return False on newlines and C-style comments.

Parameters:

line (str)

Return type:

bool

unihan_etl.core.in_fields(c, fields)

function

Return True if string is in the default fields.

Parameters:

• c (str)

• fields (Sequence[str])

Return type:

bool

unihan_etl.core.filter_manifest(files)

function

Return filtered UNIHAN_MANIFEST from list of file names.

Parameters:

files (list[str])

Return type:

UntypedUnihanData

unihan_etl.core.files_exist(path, files)

function

Return True if all files exist in specified path.

Parameters:

• path (Path)

• files (list[str])

Return type:

bool

exception unihan_etl.core.FieldNotFound

exception

Bases: Exception

Raise if field not found in file list.

__init__(field)

method

Parameters:

field (str)

Return type:

None

exception unihan_etl.core.FileNotSupported

exception

Bases: Exception

Raise if field requested is not included in current file list.

__init__(field)

method

Parameters:

field (str)

Return type:

None

unihan_etl.core.get_files(fields)

function

Return list of files required by fields. Simple dependency resolver.

Parameters:

fields (Sequence[str])

Return type:

list[str]

unihan_etl.core.get_parser()

function

Return argparse.ArgumentParser instance for CLI.

Returns:

argument parser for CLI use.

Return type:

argparse.ArgumentParser

unihan_etl.core.has_valid_zip(zip_path)

function

Return True if valid zip exists.

Parameters:

zip_path (str or pathlib.Path) – absolute path to zip

Returns:

True if valid zip exists at path

Return type:

bool

unihan_etl.core.zip_has_files(files, zip_file)

function

Return True if zip has the files inside.

Parameters:

• files (list of str) – files inside zip file

• zip_file (zipfile.ZipFile)

Returns:

True if files inside of :py:meth:`zipfile.ZipFile.namelist()

Return type:

bool

unihan_etl.core.download(url, dest, urlretrieve_fn=<function urlretrieve>, reporthook=None, cache=True)

function

Download UNIHAN zip from URL to destination.

Parameters:

• url (str or pathlib.Path) – URL to download from.

• dest (pathlib.Path) – file path where download is to be saved.

• urlretrieve_fn (UrlRetrieveFn) – function to download file

• reporthook (ReportHookFn, Optional) – Function to write progress bar to stdout buffer.

• cache (bool)

Returns:

destination where file downloaded to.

Return type:

pathlib.Path

unihan_etl.core.load_data(files)

function

Extract zip and process information into CSV’s.

Parameters:

files (list of str)

Returns:

combined data from files

Return type:

str

unihan_etl.core.extract_zip(zip_path, dest_dir)

function

Extract zip file. Return zipfile.ZipFile instance.

Parameters:

• zip_file (pathlib.Path) – filepath to extract.

• dest_dir (pathlib.Path) – directory to extract to.

• zip_path (Path)

Returns:

The extracted zip.

Return type:

zipfile.ZipFile

unihan_etl.core.normalize(raw_data, fields)

function

Return normalized data from a UNIHAN data files.

Parameters:

• raw_data (str) – combined text files from UNIHAN

• fields (list of str) – list of columns to pull

Returns:

list of unihan character information

Return type:

list

unihan_etl.core.expand_delimiters(normalized_data)

function

Return expanded multi-value fields in UNIHAN.

Parameters:

normalized_data (list of dict) – Expects data in list of hashes, per core.normalize()

Returns:

Items which have fields with delimiters and custom separation rules, will be expanded. Including multi-value fields not using both fields (so all fields stay consistent).

Return type:

list of dict

unihan_etl.core.listify(data, fields)

function

Convert tabularized data to a CSV-friendly list.

Parameters:

• data (list of dict)

• params (list of str) – keys/columns, e.g. [‘kDictionary’]

• fields (Sequence[str])

Return type:

ListifiedExport

unihan_etl.core.export_csv(data, destination, fields)

function

Export UNIHAN in flattened, CSV format.

Parameters:

• data (UntypedNormalizedData)

• destination (StrPath)

• fields (ColumnData)

Return type:

None

unihan_etl.core.export_json(data, destination)

function

Export UNIHAN in JSON format.

Parameters:

• data (UntypedNormalizedData)

• destination (StrPath)

Return type:

None

unihan_etl.core.export_yaml(data, destination)

function

Export UNIHAN in YAML format.

Parameters:

• data (UntypedNormalizedData)

• destination (StrPath)

Return type:

None

unihan_etl.core.is_default_option(field_name, val)

function

Return True if option is a unihan-etl default.

Parameters:

• field_name (str)

• val (Any)

Return type:

bool

unihan_etl.core.validate_options(options)

function

Validate unihan-etl options.

Parameters:

options (Options)

Return type:

TypeGuard[Options]

class unihan_etl.core.Packager

class

Bases: object

Download, ETL, and customize an export of UNIHAN.

UNIHAN Documentation: http://www.unicode.org/reports/tr38/

__init__(options=Options(source='http://www.unicode.org/Public/UNIDATA/Unihan.zip', destination=PosixPath('/home/runner/.local/share/unihan_etl/unihan.csv'), zip_path=PosixPath('/home/runner/.cache/unihan_etl/downloads/Unihan.zip'), work_dir=PosixPath('/home/runner/.cache/unihan_etl/downloads'), fields=('ucn', 'char', 'kAccountingNumeric', 'kAlternateTotalStrokes', 'kBigFive', 'kCCCII', 'kCNS1986', 'kCNS1992', 'kCangjie', 'kCantonese', 'kCheungBauer', 'kCheungBauerIndex', 'kCihaiT', 'kCompatibilityVariant', 'kCowles', 'kDaeJaweon', 'kDefinition', 'kEACC', 'kFanqie', 'kFenn', 'kFennIndex', 'kFourCornerCode', 'kGB0', 'kGB1', 'kGB3', 'kGB5', 'kGB8', 'kGSR', 'kGradeLevel', 'kHDZRadBreak', 'kHKGlyph', 'kHanYu', 'kHangul', 'kHanyuPinlu', 'kHanyuPinyin', 'kIBMJapan', 'kIICore', 'kIRGDaeJaweon', 'kIRGHanyuDaZidian', 'kIRGKangXi', 'kIRG_GSource', 'kIRG_HSource', 'kIRG_JSource', 'kIRG_KPSource', 'kIRG_KSource', 'kIRG_MSource', 'kIRG_SSource', 'kIRG_TSource', 'kIRG_UKSource', 'kIRG_USource', 'kIRG_VSource', 'kJIS0213', 'kJapanese', 'kJapaneseKun', 'kJapaneseOn', 'kJinmeiyoKanji', 'kJis0', 'kJis1', 'kJoyoKanji', 'kKangXi', 'kKarlgren', 'kKorean', 'kKoreanEducationHanja', 'kKoreanName', 'kLau', 'kMainlandTelegraph', 'kMandarin', 'kMatthews', 'kMeyerWempe', 'kMojiJoho', 'kMorohashi', 'kNelson', 'kOtherNumeric', 'kPhonetic', 'kPrimaryNumeric', 'kPseudoGB1', 'kRSAdobe_Japan1_6', 'kRSUnicode', 'kSBGY', 'kSMSZD2003Index', 'kSMSZD2003Readings', 'kSemanticVariant', 'kSimplifiedVariant', 'kSpecializedSemanticVariant', 'kSpoofingVariant', 'kStrange', 'kTGH', 'kTGHZ2013', 'kTaiwanTelegraph', 'kTang', 'kTayNumeric', 'kTotalStrokes', 'kTraditionalVariant', 'kUnihanCore2020', 'kVietnamese', 'kVietnameseNumeric', 'kXHC1983', 'kXerox', 'kZVariant', 'kZhuang', 'kZhuangNumeric'), format='csv', input_files=['Unihan_DictionaryIndices.txt', 'Unihan_DictionaryLikeData.txt', 'Unihan_IRGSources.txt', 'Unihan_NumericValues.txt', 'Unihan_OtherMappings.txt', 'Unihan_RadicalStrokeCounts.txt', 'Unihan_Readings.txt', 'Unihan_Variants.txt'], download=False, expand=True, prune_empty=True, cache=True, log_level='INFO'))

method

Initialize UNIHAN Packager.

Parameters:

options (dict or Options) – options values to override defaults.

Return type:

None

options: Options

attribute

download(urlretrieve_fn=<function urlretrieve>)

method

Download raw UNIHAN data if not exists.

Parameters:

urlretrieve_fn (function) – function to download file

Return type:

None

export()

method

Extract zip and process information into CSV’s.

Return type:

None | UntypedNormalizedData

classmethod from_cli(argv)

classmethod method

Create Packager instance from CLI argparse arguments.

Parameters:

argv (list) – Arguments passed in via CLI.

Returns:

builder

Return type:

Packager

unihan_etl.core.setup_logger(logger=None, level='DEBUG')

function

Configure logger for CLI use.

Parameters:

• logger (Logger) – instance of logger

• level (str) – logging level, e.g. ‘DEBUG’

Return type:

None