Skip to content

Utilities & Helpers

taters.helpers.feature_gather

AggregationPlan dataclass

AggregationPlan(
    group_by,
    per_file=True,
    stats=("mean", "std"),
    exclude_cols=(),
    include_regex=None,
    exclude_regex=None,
    dropna=False,
)

Plan describing how numeric feature columns should be aggregated.

Parameters:

Name Type Description Default
group_by Sequence[str]

One or more column names used as grouping keys (e.g., ["speaker"]).

required
per_file bool

If True, include "source" in the grouping keys to aggregate within each input file; if False, aggregate across all files globally.

True
stats Sequence[str]

Statistical reductions to compute for each numeric feature column. mean, std, var, sum, count, min and max run in one streaming pass; any other name ("median", say) is passed to pandas.DataFrame.agg and holds the table in memory.

("mean", "std")
exclude_cols Sequence[str]

Columns to drop before filtering/selecting numeric features (e.g., timestamps or free text).

()
include_regex str or None

Optional regex; if provided, only columns matching this pattern are kept (after excluding exclude_cols).

None
exclude_regex str or None

Optional regex; if provided, columns matching this pattern are removed (after applying include_regex, if any).

None
dropna bool

Whether to drop rows with NA in any of the group-by keys before grouping. The default keeps them, so rows with a missing key land in their own clearly-labeled group instead of vanishing from the output.

False
Notes

This plan is consumed by :func:aggregate_features. Column filtering happens before numeric selection; only columns that remain and can be coerced to numeric will be aggregated.

aggregate_features

aggregate_features(
    *,
    root_dir,
    pattern="*.csv",
    recursive=True,
    delimiter=",",
    encoding="utf-8-sig",
    add_source_path=False,
    plan,
    out_csv=None,
    overwrite_existing=False,
    verbose=True,
    on_progress=None
)

Discover files, read, concatenate, and aggregate numeric columns per plan.

This function consolidates tables from a single folder (or one table in a database), filters columns, coerces candidate features to numeric, groups by the specified keys, and computes the requested statistics. Output columns for aggregated features are flattened with the pattern "{column}__{stat}".

Parameters:

Name Type Description Default
root_dir PathLike

Folder containing per-item CSVs, a single CSV file, or a database table (db://<connection>/<database>/<table>).

required
pattern str

Glob pattern for selecting files.

"*.csv"
recursive bool

Recurse into subdirectories when True.

True
delimiter str

CSV delimiter.

","
encoding str

CSV encoding for read/write.

"utf-8-sig"
add_source_path bool

If True, include absolute path in "source_path" prior to filtering.

False
plan AggregationPlan

Aggregation configuration (group keys, stats, filters, NA handling).

required
out_csv PathLike or None

Output path. If None, defaults to <root_dir_parent>/<root_dir_name>.csv.

None
overwrite_existing bool

If False and out_csv exists, return it without recomputation.

False

Returns:

Type Description
Path

Path to the written CSV of aggregated features (the ref, for a database table).

Raises:

Type Description
FileNotFoundError

If no files match the pattern under root_dir.

RuntimeError

If files were found but none could be read successfully.

ValueError

If required group-by columns are missing, or if no numeric columns remain after filtering, or if per-file grouping is requested but the "source" column is absent.

Notes

Group keys are preserved as leading columns in the output. The output places "source" (and optionally "source_path") first when present. The statistics in :data:RUNNING_STATS are folded in one pass with a few numbers per group held; any other statistic goes through pandas and holds the whole table.

Source code in src\taters\helpers\feature_gather.py
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
@records_settings(binding=GATHER_BINDING, outputs=("out_csv",))
def aggregate_features(
    *,
    root_dir: PathLike,
    pattern: str = "*.csv",
    recursive: bool = True,
    delimiter: str = ",",
    encoding: str = "utf-8-sig",
    add_source_path: bool = False,
    plan: AggregationPlan,
    out_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    verbose: bool = True,
    on_progress: Optional[Callable[..., None]] = None,
) -> Path:
    """
    Discover files, read, concatenate, and aggregate numeric columns per plan.

    This function consolidates tables from a single folder (or one table in
    a database), filters columns, coerces candidate features to numeric,
    groups by the specified keys, and computes the requested statistics.
    Output columns for aggregated features are flattened with the pattern
    ``"{column}__{stat}"``.

    Parameters
    ----------
    root_dir : PathLike
        Folder containing per-item CSVs, a single CSV file, or a database
        table (``db://<connection>/<database>/<table>``).
    pattern : str, default="*.csv"
        Glob pattern for selecting files.
    recursive : bool, default=True
        Recurse into subdirectories when True.
    delimiter : str, default=","
        CSV delimiter.
    encoding : str, default="utf-8-sig"
        CSV encoding for read/write.
    add_source_path : bool, default=False
        If True, include absolute path in ``"source_path"`` prior to filtering.
    plan : AggregationPlan
        Aggregation configuration (group keys, stats, filters, NA handling).
    out_csv : PathLike or None, default=None
        Output path. If None, defaults to
        ``<root_dir_parent>/<root_dir_name>.csv``.
    overwrite_existing : bool, default=False
        If False and `out_csv` exists, return it without recomputation.

    Returns
    -------
    pathlib.Path
        Path to the written CSV of aggregated features (the ref, for a
        database table).

    Raises
    ------
    FileNotFoundError
        If no files match the pattern under `root_dir`.
    RuntimeError
        If files were found but none could be read successfully.
    ValueError
        If required group-by columns are missing,
        or if no numeric columns remain after filtering,
        or if per-file grouping is requested but the ``"source"`` column is absent.

    Notes
    -----
    Group keys are preserved as leading columns in the output. The output places
    ``"source"`` (and optionally ``"source_path"``) first when present. The
    statistics in :data:`RUNNING_STATS` are folded in one pass with a few
    numbers per group held; any other statistic goes through pandas and
    holds the whole table.
    """
    import numpy as np

    root = tables.parse_ref(root_dir)
    out_ref = _default_out(root, out_csv)

    if tables.exists(out_ref) and not overwrite_existing:
        if verbose:
            print(f"Aggregated feature output file already exists; returning existing file: {out_ref}")
        return tables.handle(out_ref)

    files = _sources(root, pattern=pattern, recursive=recursive)
    if not files:
        raise FileNotFoundError(f"No files matched {pattern} under {root}")

    announce(on_progress, "reading feature tables")
    ticker = Ticker(on_progress, len(files))
    plans = _read_plans(files, delimiter=delimiter, encoding=encoding,
                        add_source_path=add_source_path, verbose=verbose, tag="aggregate")
    if not plans:
        raise RuntimeError("No CSVs could be read successfully.")

    header = _union([[*lead, *cols] for _, lead, cols in plans])

    # if we're aggregating across files (per_file=False), we promote the inner
    # keys (e.g., 'source.1' -> 'source') and demote the file-level keys. the
    # names move; the positions don't, which is what lets each table's cells
    # still land in the right slot
    if not plan.per_file:
        header = _promote_inner_keys(header, plan.group_by)

    # build the (base) group keys from the plan
    group_keys = list(plan.group_by)
    if plan.per_file:
        if "source" not in header:
            raise ValueError("source column is missing; cannot group per_file.")
        group_keys = ["source"] + group_keys

    # resolve any collisions against the columns we actually have
    group_keys = _resolve_keys(group_keys, header)

    # now we filter, but we ALWAYS keep the group keys
    kept = _filter_columns(
        header,
        exclude_cols=tuple(plan.exclude_cols) + ("source_path",),
        include_regex=plan.include_regex,
        exclude_regex=plan.exclude_regex,
        must_keep=group_keys,
    )

    missing = [k for k in group_keys if k not in kept]
    if missing:
        raise ValueError(f"Missing group-by columns in data: {missing}")

    # everything that isn't a key is a candidate feature; which of them are
    # numeric we only know once we've seen the cells
    feature_cols = [c for c in kept if c not in set(group_keys)]
    if not feature_cols:
        raise ValueError("No numeric columns available for aggregation after filtering.")

    stats = [str(s) for s in plan.stats]
    # the promotion renamed the gathered header by position, so each table's
    # own columns have to be mapped through the header they were unioned
    # into, not through the renamed one. _group_rows does that from `plans`
    # against the pre-promotion names, so hand it those
    pre = _union([[*lead, *cols] for _, lead, cols in plans])
    rows = _group_rows(plans, pre, [pre[header.index(k)] for k in group_keys],
                       [pre[header.index(c)] for c in feature_cols],
                       delimiter=delimiter, encoding=encoding, dropna=plan.dropna,
                       ticker=ticker)

    width = len(feature_cols)
    seen = np.zeros(width, dtype=np.int64)
    if all(s in RUNNING_STATS for s in stats):
        # one pass, a few vectors per group, and the rows are gone. this is
        # what keeps a million utterance embeddings from being a million
        # rows of 768 floats in RAM
        groups: Dict[tuple, _Running] = {}
        for group, x in rows:
            acc = groups.get(group)
            if acc is None:
                acc = groups[group] = _Running(width)
            acc.add(x)
            seen += ~np.isnan(x)
        results = {g: {s: acc.stat(s) for s in stats} for g, acc in groups.items()}
    else:
        # a statistic with no running form (a median, a quantile) needs
        # every value, so these go through pandas and hold the table
        import pandas as pd

        keys_held: List[tuple] = []
        held: list = []
        for group, x in rows:
            keys_held.append(group)
            held.append(x)
            seen += ~np.isnan(x)
        values = np.vstack(held) if held else np.zeros((0, width))
        frame = pd.DataFrame(values, columns=feature_cols)
        for i, k in enumerate(group_keys):
            frame.insert(i, k, [g[i] if g[i] else np.nan for g in keys_held])
        agg = frame.groupby(group_keys, dropna=plan.dropna).agg(
            {c: stats for c in feature_cols})
        results = {}
        for key, row in agg.iterrows():
            key = key if isinstance(key, tuple) else (key,)
            group = tuple("" if isinstance(k, float) and np.isnan(k) else str(k) for k in key)
            results[group] = {s: np.array([row[(c, s)] for c in feature_cols])
                              for s in stats}

    # a column of pure text parses to nothing anywhere -- if we kept it we'd
    # spit out meaningless "<textcol>__mean" columns full of blanks. any
    # column where at least one value parsed as a number, we keep
    usable = [j for j in range(width) if seen[j] > 0]
    if not usable:
        raise ValueError("No numeric columns available for aggregation after filtering.")

    # make sure 'source' (and 'source_path' if present) lead the output
    lead = [c for c in ("source", "source_path") if c in group_keys]
    key_order = lead + [k for k in group_keys if k not in lead]
    key_pos = [group_keys.index(k) for k in key_order]
    out_header = key_order + [f"{feature_cols[j]}__{s}" for j in usable for s in stats]
    hints = {**{k: "text" for k in key_order},
             **{f"{feature_cols[j]}__{s}": ("int" if s == "count" else "float")
                for j in usable for s in stats}}

    # groups come out sorted on their keys, a blank key last, which is the
    # order a groupby wrote them in
    ordered = sorted(results, key=lambda g: tuple((not k, k) for k in g))
    with tables.write_table(out_ref, out_header, encoding=encoding, hints=hints) as writer:
        for group in ordered:
            stat = results[group]
            writer.writerow([*(group[i] for i in key_pos),
                             *(_cell(stat[s][j], s) for j in usable for s in stats)])
    return tables.handle(out_ref)

feature_gather

feature_gather(
    *,
    root_dir,
    pattern="*.csv",
    recursive=True,
    delimiter=",",
    encoding="utf-8-sig",
    add_source_path=False,
    aggregate=False,
    plan=None,
    group_by=None,
    per_file=True,
    stats=("mean", "std"),
    exclude_cols=(),
    include_regex=None,
    exclude_regex=None,
    dropna=False,
    out_csv=None,
    overwrite_existing=False,
    verbose=True,
    on_progress=None
)

Single entry point to concatenate or aggregate feature CSVs from one folder.

If aggregate=False, CSVs are concatenated with origin metadata (see :func:gather_csvs_to_one). If aggregate=True, numeric feature columns are aggregated per the provided or constructed plan (see :func:aggregate_features).

Parameters:

Name Type Description Default
root_dir PathLike

Folder containing per-item CSVs (or a single CSV file), or a table in a database (db://<connection>/<database>/<table>).

required
pattern str

Glob pattern for selecting CSV files.

"*.csv"
recursive bool

Recurse into subdirectories when True.

True
delimiter str

CSV delimiter.

","
encoding str

CSV encoding.

"utf-8-sig"
add_source_path bool

If True, include a "source_path" column in outputs.

False
aggregate bool

Toggle aggregation mode. If False, files are concatenated.

False
plan AggregationPlan or None

Explicit plan for aggregation. Required if aggregate=True and group_by is not given.

None
group_by Sequence[str] or None

Quick-plan keys. Used only when aggregate=True and plan is None.

None
per_file bool

Quick-plan flag; include "source" in grouping keys to aggregate per file.

True
stats Sequence[str]

Quick-plan statistics to compute per numeric column.

("mean", "std")
exclude_cols Sequence[str]

Quick-plan columns to drop before numeric selection.

()
include_regex str or None

Quick-plan regex to include feature columns by name.

None
exclude_regex str or None

Quick-plan regex to exclude feature columns by name.

None
dropna bool

Quick-plan NA handling for group keys. When True, rows whose group key is missing are dropped before aggregating.

False
out_csv PathLike or None

Output table. If None, defaults to <root_dir_parent>/<root_dir_name>.csv.

None
overwrite_existing bool

If False and out_csv exists, the existing path is returned without recomputation.

False

Returns:

Type Description
Path

Path to the resulting CSV (the ref, for a database table).

Raises:

Type Description
ValueError

If aggregate=True and neither plan nor group_by is provided.

See Also

gather_csvs_to_one : Concatenate CSVs with origin metadata. aggregate_features : Aggregate numeric columns according to a plan.

Source code in src\taters\helpers\feature_gather.py
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
def feature_gather(
    *,
    root_dir: PathLike,
    pattern: str = "*.csv",
    recursive: bool = True,
    delimiter: str = ",",
    encoding: str = "utf-8-sig",
    add_source_path: bool = False,
    # toggle aggregation; when True you must pass a plan (or plan_args below)
    aggregate: bool = False,
    plan: Optional[AggregationPlan] = None,
    # optional “quick plan” args (only used if plan=None and aggregate=True)
    group_by: Optional[Sequence[str]] = None,
    per_file: bool = True,
    stats: Sequence[str] = ("mean", "std"),
    exclude_cols: Sequence[str] = (),
    include_regex: Optional[str] = None,
    exclude_regex: Optional[str] = None,
    dropna: bool = False,
    # output
    out_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    verbose: bool = True,
    on_progress: Optional[Callable[..., None]] = None,
) -> Path:
    """
    Single entry point to concatenate or aggregate feature CSVs from one folder.

    If ``aggregate=False``, CSVs are concatenated with origin metadata
    (see :func:`gather_csvs_to_one`). If ``aggregate=True``, numeric feature
    columns are aggregated per the provided or constructed plan
    (see :func:`aggregate_features`).

    Parameters
    ----------
    root_dir : PathLike
        Folder containing per-item CSVs (or a single CSV file), or a table
        in a database (``db://<connection>/<database>/<table>``).
    pattern : str, default="*.csv"
        Glob pattern for selecting CSV files.
    recursive : bool, default=True
        Recurse into subdirectories when True.
    delimiter : str, default=","
        CSV delimiter.
    encoding : str, default="utf-8-sig"
        CSV encoding.
    add_source_path : bool, default=False
        If True, include a ``"source_path"`` column in outputs.
    aggregate : bool, default=False
        Toggle aggregation mode. If False, files are concatenated.
    plan : AggregationPlan or None, default=None
        Explicit plan for aggregation. Required if ``aggregate=True`` and
        ``group_by`` is not given.
    group_by : Sequence[str] or None, default=None
        Quick-plan keys. Used only when ``aggregate=True`` and ``plan`` is None.
    per_file : bool, default=True
        Quick-plan flag; include ``"source"`` in grouping keys to aggregate per file.
    stats : Sequence[str], default=("mean", "std")
        Quick-plan statistics to compute per numeric column.
    exclude_cols : Sequence[str], default=()
        Quick-plan columns to drop before numeric selection.
    include_regex : str or None, default=None
        Quick-plan regex to include feature columns by name.
    exclude_regex : str or None, default=None
        Quick-plan regex to exclude feature columns by name.
    dropna : bool, default=False
        Quick-plan NA handling for group keys. When True, rows whose group key
        is missing are dropped before aggregating.
    out_csv : PathLike or None, default=None
        Output table. If None, defaults to
        ``<root_dir_parent>/<root_dir_name>.csv``.
    overwrite_existing : bool, default=False
        If False and `out_csv` exists, the existing path is returned without
        recomputation.

    Returns
    -------
    pathlib.Path
        Path to the resulting CSV (the ref, for a database table).

    Raises
    ------
    ValueError
        If ``aggregate=True`` and neither ``plan`` nor ``group_by`` is provided.

    See Also
    --------
    gather_csvs_to_one : Concatenate CSVs with origin metadata.
    aggregate_features : Aggregate numeric columns according to a plan.
    """

    if not aggregate:
        return gather_csvs_to_one(
            on_progress=on_progress,
            root_dir=root_dir,
            pattern=pattern,
            recursive=recursive,
            delimiter=delimiter,
            encoding=encoding,
            add_source_path=add_source_path,
            out_csv=out_csv,
            overwrite_existing=overwrite_existing,
            verbose=verbose,
        )

    # aggregate=True
    if plan is None:
        if not group_by:
            raise ValueError("When aggregate=True, you must provide 'plan' or 'group_by'.")
        plan = make_plan(
            group_by=group_by,
            per_file=per_file,
            stats=stats,
            exclude_cols=exclude_cols,
            include_regex=include_regex,
            exclude_regex=exclude_regex,
            dropna=dropna,
        )

    return aggregate_features(
        on_progress=on_progress,
        root_dir=root_dir,
        pattern=pattern,
        recursive=recursive,
        delimiter=delimiter,
        encoding=encoding,
        add_source_path=add_source_path,
        plan=plan,
        out_csv=out_csv,
        overwrite_existing=overwrite_existing,
        verbose=verbose,
    )

gather_csvs_to_one

gather_csvs_to_one(
    *,
    root_dir,
    pattern="*.csv",
    recursive=True,
    delimiter=",",
    encoding="utf-8-sig",
    add_source_path=False,
    out_csv=None,
    overwrite_existing=False,
    verbose=True,
    on_progress=None
)

Concatenate many CSVs into a single CSV with origin metadata.

Each input table is streamed through, a leading "source" column is inserted (and optionally "source_path"), and rows are appended. The final table's columns are the union of every input's, in order of first appearance, with "source" (and, if present, "source_path") leading.

Parameters:

Name Type Description Default
root_dir PathLike

Folder containing CSVs, a single CSV file, or a database table.

required
pattern str

Glob pattern for selecting files.

"*.csv"
recursive bool

Recurse into subdirectories when True.

True
delimiter str

CSV delimiter.

","
encoding str

CSV encoding for read/write.

"utf-8-sig"
add_source_path bool

If True, include absolute path in "source_path".

False
out_csv PathLike or None

Output path. If None, defaults to <root_dir_parent>/<root_dir_name>.csv.

None
overwrite_existing bool

If False and out_csv exists, return it without recomputation.

False

Returns:

Type Description
Path

Path to the written CSV (the ref, for a database table).

Raises:

Type Description
FileNotFoundError

If no files match the pattern under root_dir.

RuntimeError

If files were found but none could be read successfully.

Notes

Cells are copied as the text they are. Column order from inputs is preserved after the leading origin columns.

Source code in src\taters\helpers\feature_gather.py
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
@records_settings(binding=GATHER_BINDING, outputs=("out_csv",))
def gather_csvs_to_one(
    *,
    root_dir: PathLike,
    pattern: str = "*.csv",
    recursive: bool = True,
    delimiter: str = ",",
    encoding: str = "utf-8-sig",
    add_source_path: bool = False,
    out_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    verbose: bool = True,
    on_progress: Optional[Callable[..., None]] = None,
) -> Path:
    """
    Concatenate many CSVs into a single CSV with origin metadata.

    Each input table is streamed through, a leading ``"source"`` column is
    inserted (and optionally ``"source_path"``), and rows are appended. The
    final table's columns are the union of every input's, in order of first
    appearance, with ``"source"`` (and, if present, ``"source_path"``)
    leading.

    Parameters
    ----------
    root_dir : PathLike
        Folder containing CSVs, a single CSV file, or a database table.
    pattern : str, default="*.csv"
        Glob pattern for selecting files.
    recursive : bool, default=True
        Recurse into subdirectories when True.
    delimiter : str, default=","
        CSV delimiter.
    encoding : str, default="utf-8-sig"
        CSV encoding for read/write.
    add_source_path : bool, default=False
        If True, include absolute path in ``"source_path"``.
    out_csv : PathLike or None, default=None
        Output path. If None, defaults to
        ``<root_dir_parent>/<root_dir_name>.csv``.
    overwrite_existing : bool, default=False
        If False and `out_csv` exists, return it without recomputation.

    Returns
    -------
    pathlib.Path
        Path to the written CSV (the ref, for a database table).

    Raises
    ------
    FileNotFoundError
        If no files match the pattern under `root_dir`.
    RuntimeError
        If files were found but none could be read successfully.

    Notes
    -----
    Cells are copied as the text they are. Column order from inputs is
    preserved after the leading origin columns.
    """

    root = tables.parse_ref(root_dir)
    out_ref = _default_out(root, out_csv)

    if tables.exists(out_ref) and not overwrite_existing:
        if verbose:
            print(f"Aggregated feature output file already exists; returning existing file: {out_ref}")
        return tables.handle(out_ref)

    files = _sources(root, pattern=pattern, recursive=recursive)
    if not files:
        raise FileNotFoundError(f"No files matched {pattern} under {root}")

    # we tick once per table read: a big embeddings folder took a minute under
    # a bare spinner, which looks an awful lot like a hang.
    announce(on_progress, "reading feature tables")
    ticker = Ticker(on_progress, len(files))
    plans = _read_plans(files, delimiter=delimiter, encoding=encoding,
                        add_source_path=add_source_path, verbose=verbose, tag="gather")
    if not plans:
        raise RuntimeError("No CSVs could be read successfully.")

    # the headers decide the shape; the rows then stream straight through
    # from each table into the one output, so nothing is held but the row
    # in hand. make sure 'source' is first (and 'source_path' next, if we have it)
    lead = ["source"] + (["source_path"] if add_source_path else [])
    header = lead + [c for c in _union([[*ld, *cols] for _, ld, cols in plans])
                     if c not in lead]
    at = {c: i for i, c in enumerate(header)}
    with tables.write_table(out_ref, header, encoding=encoding,
                            hints={c: "text" for c in lead}) as writer:
        for ref, ld, cols in plans:
            origin = _origin_values(ref, ld)
            slots = [at[c] for c in cols]
            lead_slots = [at[c] for c in ld]
            for cells in tables.read_cells(ref, encoding=encoding, delimiter=delimiter):
                row = [""] * len(header)
                for i, v in zip(lead_slots, origin):
                    row[i] = v
                for i, v in zip(slots, cells):
                    row[i] = v
                writer.writerow(row)
            ticker.tick(message="reading feature tables")
    return tables.handle(out_ref)

make_plan

make_plan(
    *,
    group_by,
    per_file=True,
    stats=("mean", "std"),
    exclude_cols=(),
    include_regex=None,
    exclude_regex=None,
    dropna=False
)

Create an :class:AggregationPlan from simple arguments.

Parameters:

Name Type Description Default
group_by Sequence[str]

Grouping key(s) to use (e.g., ["speaker"]).

required
per_file bool

If True, group within files by including "source" in group keys.

True
stats Sequence[str]

Statistical reductions to compute per numeric column.

("mean", "std")
exclude_cols Sequence[str]

Columns to drop prior to feature selection.

()
include_regex str or None

Regex to include feature columns by name.

None
exclude_regex str or None

Regex to exclude feature columns by name.

None
dropna bool

Drop rows with NA in any group key. Off by default so rows with a missing key are still reported rather than silently discarded.

False

Returns:

Type Description
AggregationPlan

A configured plan instance for :func:aggregate_features.

Source code in src\taters\helpers\feature_gather.py
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
def make_plan(
    *,
    group_by: Sequence[str],
    per_file: bool = True,
    stats: Sequence[str] = ("mean", "std"),
    exclude_cols: Sequence[str] = (),
    include_regex: Optional[str] = None,
    exclude_regex: Optional[str] = None,
    dropna: bool = False,
) -> AggregationPlan:
    """
    Create an :class:`AggregationPlan` from simple arguments.

    Parameters
    ----------
    group_by : Sequence[str]
        Grouping key(s) to use (e.g., ``["speaker"]``).
    per_file : bool, default=True
        If True, group within files by including ``"source"`` in group keys.
    stats : Sequence[str], default=("mean", "std")
        Statistical reductions to compute per numeric column.
    exclude_cols : Sequence[str], default=()
        Columns to drop prior to feature selection.
    include_regex : str or None, default=None
        Regex to include feature columns by name.
    exclude_regex : str or None, default=None
        Regex to exclude feature columns by name.
    dropna : bool, default=False
        Drop rows with NA in any group key. Off by default so rows with a
        missing key are still reported rather than silently discarded.

    Returns
    -------
    AggregationPlan
        A configured plan instance for :func:`aggregate_features`.
    """

    return AggregationPlan(
        group_by=tuple(group_by),
        per_file=per_file,
        stats=tuple(stats),
        exclude_cols=tuple(exclude_cols),
        include_regex=include_regex,
        exclude_regex=exclude_regex,
        dropna=dropna,
    )

taters.helpers.find_files

find_files

find_files(
    root_dir,
    *,
    file_type="video",
    extensions=None,
    recursive=True,
    follow_symlinks=False,
    include_hidden=False,
    include_globs=None,
    exclude_globs=None,
    absolute=True,
    sort=True,
    ffprobe_verify=False
)

Discover media files under a folder using smart, FFmpeg-friendly filters.

You can either (a) choose a built-in group of extensions via file_type ("audio"|"video"|"image"|"subtitle"|"archive"|"any") or (b) pass an explicit list of extensions to match. Matching is case-insensitive; dots are optional (e.g., ".wav" and "wav" are equivalent). Hidden files and directories are excluded by default.

For audio/video, ffprobe_verify=True additionally checks that at least one corresponding stream is present (e.g., exclude MP4s with no audio when file_type="audio"). This is slower but robust when your dataset contains “container only” files.

Parameters:

Name Type Description Default
root_dir str | PathLike

Folder to scan.

required
file_type str

Built-in group selector. Ignored if extensions is provided.

'video'
extensions Optional[Sequence[str]]

Explicit extensions to include (e.g., [".wav",".flac"]). Overrides file_type.

None
recursive bool

Recurse into subfolders. Default: True.

True
follow_symlinks bool

Follow directory symlinks during traversal. Default: False.

False
include_hidden bool

Include dot-files and dot-dirs. Default: False.

False
include_globs Optional[Sequence[str]]

Additional glob filters applied after extension filtering; include_globs uses OR-semantics, then exclude_globs removes matches.

None
absolute bool

Return absolute paths when True (default) else relative to root_dir.

True
sort bool

Sort lexicographically (case-insensitive). Default: True.

True
ffprobe_verify bool

For audio/video, keep only files where ffprobe reports ≥1 matching stream.

False

Returns:

Type Description
list[Path]

The matched files.

Raises:

Type Description
FileNotFoundError

If root_dir does not exist.

ValueError

If file_type is not one of the supported groups.

Examples:

Find all videos (recursive), as absolute paths:

>>> find_files("dataset", file_type="video")

Use explicit extensions and keep paths relative:

>>> find_files("dataset", extensions=[".wav",".flac"], absolute=False)

Only include files matching a glob and exclude temp folders:

>>> find_files("dataset", file_type="audio",
...            include_globs=["**/*session*"], exclude_globs=["**/tmp/**"])

Verify playable audio streams exist:

>>> find_files("dataset", file_type="audio", ffprobe_verify=True)
Source code in src\taters\helpers\find_files.py
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
def find_files(
    root_dir: str | os.PathLike,
    *,
    file_type: str = "video",                            # 'audio' | 'video' | 'image' | 'subtitle' | 'archive' | 'any'
    extensions: Optional[Sequence[str]] = None,     # explicit extensions override group (e.g., ['.wav','.flac'])
    recursive: bool = True,
    follow_symlinks: bool = False,
    include_hidden: bool = False,
    include_globs: Optional[Sequence[str]] = None,  # e.g., ['**/*session*']
    exclude_globs: Optional[Sequence[str]] = None,  # e.g., ['**/temp/**']
    absolute: bool = True,
    sort: bool = True,
    ffprobe_verify: bool = False,                   # confirm stream presence via ffprobe (audio/video only)
) -> List[Path]:
    """
    Discover media files under a folder using smart, FFmpeg-friendly filters.

    You can either (a) choose a built-in **group** of extensions via `file_type`
    (`"audio"|"video"|"image"|"subtitle"|"archive"|"any"`) or (b) pass an explicit
    list of `extensions` to match. Matching is case-insensitive; dots are optional
    (e.g., `".wav"` and `"wav"` are equivalent). Hidden files and directories are
    excluded by default.

    For audio/video, `ffprobe_verify=True` additionally checks that at least one
    corresponding stream is present (e.g., exclude MP4s with no audio when
    `file_type="audio"`). This is slower but robust when your dataset contains
    “container only” files.

    Parameters
    ----------
    root_dir
        Folder to scan.
    file_type
        Built-in group selector. Ignored if `extensions` is provided.
    extensions
        Explicit extensions to include (e.g., `[".wav",".flac"]`). Overrides `file_type`.
    recursive
        Recurse into subfolders. Default: `True`.
    follow_symlinks
        Follow directory symlinks during traversal. Default: `False`.
    include_hidden
        Include dot-files and dot-dirs. Default: `False`.
    include_globs / exclude_globs
        Additional glob filters applied after extension filtering; `include_globs`
        uses OR-semantics, then `exclude_globs` removes matches.
    absolute
        Return absolute paths when `True` (default) else relative to `root_dir`.
    sort
        Sort lexicographically (case-insensitive). Default: `True`.
    ffprobe_verify
        For `audio`/`video`, keep only files where `ffprobe` reports ≥1 matching
        stream.

    Returns
    -------
    list[pathlib.Path]
        The matched files.

    Raises
    ------
    FileNotFoundError
        If `root_dir` does not exist.
    ValueError
        If `file_type` is not one of the supported groups.

    Examples
    --------
    Find all videos (recursive), as absolute paths:

    >>> find_files("dataset", file_type="video")

    Use explicit extensions and keep paths relative:

    >>> find_files("dataset", extensions=[".wav",".flac"], absolute=False)

    Only include files matching a glob and exclude temp folders:

    >>> find_files("dataset", file_type="audio",
    ...            include_globs=["**/*session*"], exclude_globs=["**/tmp/**"])

    Verify playable audio streams exist:

    >>> find_files("dataset", file_type="audio", ffprobe_verify=True)
    """
    root_dir = Path(root_dir)
    if not root_dir.exists():
        raise FileNotFoundError(f"Root path not found: {root_dir}")

    if extensions:
        allowed = {_norm_ext(e) for e in extensions}
    else:
        if file_type not in GROUPS:
            raise ValueError(f"Unknown kind '{file_type}'. Choose from {', '.join(GROUPS.keys())}.")
        allowed = set(GROUPS[file_type])

    cand = (
        p for p in _iter_files(root_dir, recursive=recursive, follow_symlinks=follow_symlinks, include_hidden=include_hidden)
        if p.is_file() and _match_ext(p, allowed)
    )

    cand = _glob_filter(
        cand,
        includes=include_globs or [],
        excludes=exclude_globs or [],
    )

    out: List[Path] = []
    for p in cand:
        if ffprobe_verify and file_type in ("audio", "video"):
            if not _ffprobe_has_stream(p, file_type):
                continue
        out.append(p.resolve() if absolute else p)

    if sort:
        out.sort(key=lambda x: str(x).lower())
    return out

taters.helpers.text_gather

csv_to_analysis_ready_csv

csv_to_analysis_ready_csv(
    *,
    csv_path,
    out_csv=None,
    overwrite_existing=False,
    text_cols=None,
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=None,
    encoding=DEFAULT_ENCODING,
    joiner=DEFAULT_JOINER,
    num_buckets=1024,
    max_open_bucket_files=64,
    tmp_root=None,
    include_id_cols=True,
    carry_cols=None,
    agg_cols=None,
    row_filters=None,
    verbose=True,
    on_progress=None
)

Stream a (possibly huge) CSV into a compact analysis-ready CSV with a stable schema and optional external grouping.

Output schema

Always writes a header and enforces a consistent column order:

• No grouping: text_id,text (plus source_col if mode="separate") • With grouping: text_id,text,group_count (plus source_col if mode="separate")

carry_cols inserts the named source columns between the identifiers and text in both shapes.

Where: - text_id is either the composed ID from id_cols or row_<n> when id_cols=None. - mode="concat" joins all text_cols using joiner per row or group. - mode="separate" emits one row per (row_or_group, text_col) and fills source_col with the contributing column name.

Grouping at scale

If group_by is provided, the function performs a two-pass external grouping that does not require presorting: 1) Hash-partition rows to on-disk “bucket” CSVs (bounded writers with LRU). 2) Aggregate each bucket into final rows (concat or separate mode), writing group_count to record how many pieces contributed.

Parameters:

Name Type Description Default
csv_path PathLike

Source CSV with at least the columns in text_cols (and group_by if grouping).

required
out_csv PathLike | None

Destination CSV. If None, a name is derived from the input and options (e.g., <stem>_grouped_<group_by>.csv or <stem>_concat_<cols>.csv).

None
overwrite_existing bool

If False (default) and out_csv exists, the function returns early.

False
text_cols Sequence[str] | None

Text fields to concatenate or emit separately. May be empty or None: the spreadsheet is then wrangled without any text -- rows are still combined, counted, summarized and carried -- and the output has no text column.

None
id_cols Sequence[str] | None

Optional columns to compose text_id when not grouping. When omitted, a synthetic row_<n> is used.

None
carry_cols Sequence[str] | None

Source columns to copy through to the output untouched. Distinct from id_cols, which compose text_id and so cannot be used to carry a column whose values repeat -- naming speaker there would give every utterance by one person the same text_id. Downstream analyzers offer a pass_through_cols argument that is meaningless unless the gather that feeds them preserved the columns, which is what this is for.

When grouping, a carried column is written only where its value is the same for every row in the group; where rows disagree the cell is blank, because there is no honest single answer and picking the first row's is the kind of quiet wrong answer that survives into a published table.

Names in carry_cols that the source does not have are dropped with a warning rather than raising: unlike text_cols, a carried column is a convenience, and half the presets that ask for speaker run over inputs that never had one.

None
agg_cols Mapping[str, str] | Sequence[str] | None

Number columns to summarize per group, e.g. a per-post score becoming the average score of everything a user wrote. A sequence of names means "the mean of each"; a mapping picks the statistic per column from mean, sum, min and max. Each summary lands in its own output column named <col>_<stat>, placed after group_count.

Values that are blank or not parseable as numbers are skipped rather than treated as zero -- a missing score is missing, not 0 -- and a group with no numeric values at all gets a blank cell, for the same NA-is-not-zero reason a disagreeing carried column does.

None
row_filters Sequence[Sequence[object]] | None

Which rows to keep, as [[column, operator, value], ...] over the source spreadsheet's own columns -- [["age", ">=", 18]], [["condition", "in", ["A", "B"]]]. A row has to clear all of them. The operators and the comparison are :mod:taters.helpers.row_filter's, the same ones the statistics use, so a filter means one thing wherever it is applied.

Only the spreadsheet's own columns, because nothing has been measured yet: a word count is not available here and does not belong here either. What somebody knows at this point is what they collected -- a screener somebody failed, a condition they are not analyzing, an age below the one they meant to study.

It matters most when rows are being combined, since a row inside somebody else's joined text cannot be taken out again afterwards; that is why it is applied before the rows are bucketed rather than filtered later.

Summaries run over every row of the group, including rows whose text cells are empty -- group_count counts only the rows whose text was joined, so the two can disagree. To keep that visible, each summary is followed by a <col>_n column giving the number of numeric values it actually used.

Requires group_by: without groups there is nothing to summarize over, and silently ignoring the ask would hide a real mistake. Unlike carry_cols, a name the source lacks raises: a summary is an explicit computation, not a convenience.

None
mode str

"concat" (default) or "separate". See schema above.

'concat'
group_by Sequence[str] | None

Optional list of columns to aggregate by; works on unsorted CSVs.

None
delimiter str | None

Parsing/formatting options. If delimiter=None, sniffs from a sample.

None
encoding str | None

Parsing/formatting options. If delimiter=None, sniffs from a sample.

None
joiner str | None

Parsing/formatting options. If delimiter=None, sniffs from a sample.

None
num_buckets int

External grouping controls (partition count, LRU limit, temp root).

1024
max_open_bucket_files int

External grouping controls (partition count, LRU limit, temp root).

1024
tmp_root int

External grouping controls (partition count, LRU limit, temp root).

1024
include_id_cols bool

When not grouping, write the id columns beside text_id. Grouped output carries the grouping columns instead, whatever this says.

True
on_progress

Optional (done, total, message) callback. The gather names each phase as it runs -- counting the rows, then copying them (or, when grouping, sorting them into groups and then combining each group) -- because a silent spinner over a large file reads as a hang.

None

Returns:

Type Description
Path

Path to the analysis-ready CSV.

Raises:

Type Description
ValueError

If required columns are missing or mode is invalid.

Examples:

Concatenate two text fields per row:

>>> csv_to_analysis_ready_csv(
...     csv_path="transcripts.csv",
...     text_cols=["prompt","response"],
...     id_cols=["speaker"],
... )

Group by speaker and join rows:

>>> csv_to_analysis_ready_csv(
...     csv_path="transcripts.csv",
...     text_cols=["text"],
...     group_by=["speaker"],
... )
Source code in src\taters\helpers\text_gather.py
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
@records_settings(
    # which rows and columns we read, and which metadata rode along.
    # `joiner` is NOT in here on purpose: joining two text columns with
    # " " rather than "\n" changes sentence segmentation, so it decides
    # every number every downstream analyzer produces.
    binding=("csv_path", "text_cols", "id_cols", "delimiter",
             "encoding", "include_id_cols", "carry_cols", "agg_cols"),
    # this takes seconds to redo, and it's the one table every join depends
    # on: an existing copy with no record gets rebuilt rather than trusted.
    redo_without_record=True,
    grain=TEXT_GRAIN,
    outputs=("out_csv",),
    # this step decides what text EXISTS. its settings describe the
    # dataset, so we record them but keep them out of the chain that
    # downstream steps compare -- a model has to be applicable to a corpus
    # assembled differently, which is the whole point of saving one.
    defines_text=True)
def csv_to_analysis_ready_csv(
    *,
    csv_path: PathLike,
    out_csv: PathLike | None = None,
    overwrite_existing: bool = False,
    text_cols: Sequence[str] | None = None,
    id_cols: Sequence[str] | None = None,
    mode: str = "concat",
    group_by: Sequence[str] | None = None,
    delimiter: str | None = None,
    encoding: str = DEFAULT_ENCODING,
    joiner: str = DEFAULT_JOINER,
    num_buckets: int = 1024,
    max_open_bucket_files: int = 64,
    tmp_root: PathLike | None = None,
    include_id_cols: bool = True,
    carry_cols: Sequence[str] | None = None,
    agg_cols: Mapping[str, str] | Sequence[str] | None = None,
    row_filters: Sequence[Sequence[object]] | None = None,
    verbose: bool = True,
    on_progress=None,   # on_progress(done, total, message) per row / bucket
) -> Path:
    """
    Stream a (possibly huge) CSV into a compact **analysis-ready** CSV with a
    stable schema and optional external grouping.

    Output schema
    -------------
    Always writes a header and enforces a consistent column order:

    • No grouping:
        `text_id,text`                            (plus `source_col` if `mode="separate"`)
    • With grouping:
        `text_id,text,group_count`                (plus `source_col` if `mode="separate"`)

    `carry_cols` inserts the named source columns between the identifiers and
    `text` in both shapes.

    Where:
      - `text_id` is either the composed ID from `id_cols` or `row_<n>` when
        `id_cols=None`.
      - `mode="concat"` joins all `text_cols` using `joiner` per row or group.
      - `mode="separate"` emits one row per (`row_or_group`, `text_col`) and
        fills `source_col` with the contributing column name.

    Grouping at scale
    -----------------
    If `group_by` is provided, the function performs a **two-pass external
    grouping** that does not require presorting:
      1) Hash-partition rows to on-disk “bucket” CSVs (bounded writers with LRU).
      2) Aggregate each bucket into final rows (concat or separate mode), writing
         `group_count` to record how many pieces contributed.

    Parameters
    ----------
    csv_path
        Source CSV with at least the columns in `text_cols` (and `group_by` if
        grouping).
    out_csv
        Destination CSV. If `None`, a name is derived from the input and options
        (e.g., `<stem>_grouped_<group_by>.csv` or `<stem>_concat_<cols>.csv`).
    overwrite_existing
        If `False` (default) and `out_csv` exists, the function returns early.
    text_cols
        Text fields to concatenate or emit separately. May be empty or None:
        the spreadsheet is then wrangled without any text -- rows are still
        combined, counted, summarized and carried -- and the output has no
        ``text`` column.
    id_cols
        Optional columns to compose `text_id` when not grouping. When omitted, a
        synthetic `row_<n>` is used.
    carry_cols
        Source columns to copy through to the output untouched. Distinct from
        `id_cols`, which *compose* `text_id` and so cannot be used to carry a
        column whose values repeat -- naming `speaker` there would give every
        utterance by one person the same `text_id`. Downstream analyzers offer a
        `pass_through_cols` argument that is meaningless unless the gather that
        feeds them preserved the columns, which is what this is for.

        When grouping, a carried column is written only where its value is the
        same for every row in the group; where rows disagree the cell is blank,
        because there is no honest single answer and picking the first row's is
        the kind of quiet wrong answer that survives into a published table.

        Names in `carry_cols` that the source does not have are dropped with a
        warning rather than raising: unlike `text_cols`, a carried column is a
        convenience, and half the presets that ask for `speaker` run over inputs
        that never had one.
    agg_cols
        Number columns to summarize per group, e.g. a per-post score becoming
        the average score of everything a user wrote. A sequence of names
        means "the mean of each"; a mapping picks the statistic per column
        from ``mean``, ``sum``, ``min`` and ``max``. Each summary lands in
        its own output column named ``<col>_<stat>``, placed after
        ``group_count``.

        Values that are blank or not parseable as numbers are skipped rather
        than treated as zero -- a missing score is missing, not 0 -- and a
        group with no numeric values at all gets a blank cell, for the same
        NA-is-not-zero reason a disagreeing carried column does.
    row_filters
        Which rows to keep, as ``[[column, operator, value], ...]`` over the
        source spreadsheet's own columns -- ``[["age", ">=", 18]]``,
        ``[["condition", "in", ["A", "B"]]]``. A row has to clear all of
        them. The operators and the comparison are
        :mod:`taters.helpers.row_filter`'s, the same ones the statistics use,
        so a filter means one thing wherever it is applied.

        Only the spreadsheet's own columns, because nothing has been measured
        yet: a word count is not available here and does not belong here
        either. What somebody knows at this point is what they collected --
        a screener somebody failed, a condition they are not analyzing, an
        age below the one they meant to study.

        It matters most when rows are being combined, since a row inside
        somebody else's joined text cannot be taken out again afterwards;
        that is why it is applied before the rows are bucketed rather than
        filtered later.

        Summaries run over **every** row of the group, including rows whose
        text cells are empty -- `group_count` counts only the rows whose
        text was joined, so the two can disagree. To keep that visible, each
        summary is followed by a ``<col>_n`` column giving the number of
        numeric values it actually used.

        Requires `group_by`: without groups there is nothing to summarize
        over, and silently ignoring the ask would hide a real mistake.
        Unlike `carry_cols`, a name the source lacks raises: a summary is an
        explicit computation, not a convenience.
    mode
        `"concat"` (default) or `"separate"`. See schema above.
    group_by
        Optional list of columns to aggregate by; works on unsorted CSVs.
    delimiter, encoding, joiner
        Parsing/formatting options. If `delimiter=None`, sniffs from a sample.
    num_buckets, max_open_bucket_files, tmp_root
        External grouping controls (partition count, LRU limit, temp root).
    include_id_cols
        When not grouping, write the id columns beside ``text_id``. Grouped
        output carries the grouping columns instead, whatever this says.
    on_progress
        Optional ``(done, total, message)`` callback. The gather names each
        phase as it runs -- counting the rows, then copying them (or, when
        grouping, sorting them into groups and then combining each group) --
        because a silent spinner over a large file reads as a hang.

    Returns
    -------
    Path
        Path to the analysis-ready CSV.

    Raises
    ------
    ValueError
        If required columns are missing or `mode` is invalid.

    Examples
    --------
    Concatenate two text fields per row:

    >>> csv_to_analysis_ready_csv(
    ...     csv_path="transcripts.csv",
    ...     text_cols=["prompt","response"],
    ...     id_cols=["speaker"],
    ... )

    Group by speaker and join rows:

    >>> csv_to_analysis_ready_csv(
    ...     csv_path="transcripts.csv",
    ...     text_cols=["text"],
    ...     group_by=["speaker"],
    ... )
    """
    keep = [list(f) for f in (row_filters or [])]
    spec = _db_source(csv_path)
    if spec is not None:
        return _gather_from_db(
            spec, out_csv=out_csv, overwrite_existing=overwrite_existing,
            text_cols=list(text_cols or []), id_cols=list(id_cols or []),
            mode=mode, group_by=list(group_by or []), joiner=joiner,
            include_id_cols=include_id_cols, carry_cols=list(carry_cols or []),
            agg_cols=agg_cols, keep=keep, verbose=verbose, on_progress=on_progress)
    in_path = _ensure_path(csv_path)

    # sniff out the delimiter if we weren't given one
    if delimiter is None:
        with in_path.open("rb") as fb:
            sample = fb.read(8192)
        delimiter = _detect_delimiter(sample, default=DEFAULT_DELIM)

    # empty text_cols is something people actually ask for, not a mistake: a
    # spreadsheet without text can still be combined and summarized (one row
    # per person, with average scores). the output just has no `text` column.
    text_cols = list(text_cols or [])
    mode = mode.strip().lower()
    if mode not in ("concat", "separate"):
        raise ValueError("mode must be 'concat' or 'separate'")
    if mode == "separate" and not text_cols:
        raise ValueError("mode='separate' splits rows by text column, which needs text_cols")

    # normalize agg_cols to {column: stat}. a bare sequence means "the mean of
    # each", because the average per group is almost always what people want.
    if agg_cols and not group_by:
        raise ValueError("agg_cols requires group_by: there are no groups to summarize over")
    agg: Dict[str, str] = (dict(agg_cols) if isinstance(agg_cols, Mapping)
                           else {c: "mean" for c in (agg_cols or [])})
    bad_stats = {s for s in agg.values() if s not in _AGG_STATS}
    if bad_stats:
        raise ValueError(f"agg_cols statistics must be one of {sorted(_AGG_STATS)}, got {sorted(bad_stats)}")

    include_source_col = (mode == "separate")
    include_source_path = False  # we're doing CSV here; the folder variant uses this

    # we resolve these against the real header before either path runs, so the
    # two paths can't disagree about which columns exist.
    carry: List[str] = list(carry_cols or [])
    if carry:
        with in_path.open("r", newline="", encoding=encoding) as f:
            available = csv.DictReader(f, delimiter=delimiter).fieldnames or []
        absent = [c for c in carry if c not in available]
        if absent:
            if verbose:
                print(f"[text-gather] WARNING: cannot carry columns not in the input: {absent}")
            carry = [c for c in carry if c not in absent]

    # figure out where the output goes (next to the input, unless told otherwise).
    # a table in a database is as good a place as a file: the run decides
    from . import tables

    out_path = tables.parse_ref(out_csv) if out_csv is not None else tables.parse_ref(
        _default_csv_out_path(in_csv=in_path, mode=mode, text_cols=text_cols,
                              group_by=group_by))
    if not out_path.is_db:
        out_path.path.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and tables.exists(out_path):
        if verbose:
            print("File with gathered text already exists; returning existing file.")
        return tables.handle(out_path)

    # no grouping? then we can just stream straight through to the output
    if not group_by:
        writer, fh, _ = _open_out_csv(
            out_path,
            include_source_col,
            include_source_path,
            include_group_count=False,
            id_col_names=(list(id_cols) if include_id_cols and id_cols else None),
            group_by_names=None,
            carry_col_names=carry,
            include_text=bool(text_cols),
        )
        carried = _carried_names(carry, ["text_id"] + (
            _emit_names(id_cols) if include_id_cols and id_cols else []))
        announce(on_progress, "counting rows")
        ticker = Ticker(on_progress,
                        count_rows(in_path, on_progress=on_progress,
                                   encoding=encoding))
        try:
            with in_path.open("r", newline="", encoding=encoding) as f:
                rdr = csv.DictReader(f, delimiter=delimiter)
                headers = rdr.fieldnames or []
                missing = [c for c in (id_cols or []) + text_cols if c not in headers]
                if missing:
                    raise ValueError(f"Missing columns: {missing}. Make sure that you try specifying a delimiter manually if you see this error message.")

                for idx, row in enumerate(rdr, start=1):
                    ticker.tick(message="copying rows")
                    # before `text_id` is used but after it is numbered, so a
                    # row left out here does not renumber the rows after it
                    # -- every other table in the run is keyed on that
                    # number, and shifting it would join the wrong rows to
                    # the wrong groups while looking perfectly reasonable.
                    if keep and not keeps_row(row, keep):
                        continue
                    text_id = _compose_id([row.get(c, "") for c in (id_cols or [])]) if id_cols else f"row_{idx}"
                    if mode == "concat":
                        parts = [row.get(c, "") for c in text_cols if row.get(c, "")]
                        # if we have text columns, a row with nothing in any of
                        # them has nothing to give us; if we have none, every
                        # row counts on its own.
                        if text_cols and not parts:
                            continue
                        row_prefix = [text_id]
                        if include_id_cols and id_cols:
                            row_prefix += [row.get(c, "") for c in _emit_names(id_cols)]
                        row_prefix += [row.get(c, "") for c in carried]
                        writer.writerow(row_prefix + ([joiner.join(parts)] if text_cols else []))
                    else:
                        for col in text_cols:
                            val = row.get(col, "")
                            if not val:
                                continue
                            row_prefix = [text_id]
                            if include_id_cols and id_cols:
                                row_prefix += [row.get(c, "") for c in _emit_names(id_cols)]
                            row_prefix += [row.get(c, "") for c in carried]
                            writer.writerow(row_prefix + [val, col])

        except BaseException:
            fh.close(ok=False)
            raise
        else:
            fh.close(ok=True)
        return tables.handle(out_path)

    # otherwise we do the grouping on disk, in two passes
    group_by = list(group_by)

    # pass 1: partition the rows into hash buckets
    tmp_base = Path(tempfile.mkdtemp(prefix="gather_partitions_", dir=str(tmp_root) if tmp_root else None))
    part_dir = tmp_base / "parts"
    part_dir.mkdir(parents=True, exist_ok=True)

    # the bucket writer cache
    carried = _carried_names(carry, ["text_id"] + _emit_names(group_by))
    # the columns we're aggregating ride along in the partitions too (deduped:
    # it's fine to summarize a column that's also grouped or carried, and the
    # DictReader in pass 2 needs each name to show up exactly once).
    agg_extra = [c for c in agg if c not in set(group_by) | set(carried) | set(text_cols)]
    header_small = group_by + carried + agg_extra + text_cols
    cache = _LRUFileCache(
        max_open=max_open_bucket_files,
        newline="",
        encoding=encoding,
        delimiter=delimiter,
    )


    announce(on_progress, "counting rows")
    ticker = Ticker(on_progress,
                    count_rows(in_path, on_progress=on_progress,
                               encoding=encoding))
    try:
        with in_path.open("r", newline="", encoding=encoding) as f:
            rdr = csv.DictReader(f, delimiter=delimiter)
            headers = rdr.fieldnames or []
            missing = [c for c in group_by + list(agg) + text_cols if c not in headers]
            if missing:
                raise ValueError(f"Missing columns: {missing}. Make sure that you try specifying a delimiter manually if you see this error message.")

            for row in rdr:
                ticker.tick(message="sorting rows into groups (pass 1 of 2)")
                if keep and not keeps_row(row, keep):
                    # dropped before it is bucketed, on purpose: once a row
                    # is inside somebody else's joined text there is no way
                    # to take it back out again.
                    continue
                key_tuple = tuple(row[g] for g in group_by)
                bucket = _bucket_of_key(key_tuple, num_buckets)
                bpath = part_dir / f"bucket_{bucket:05d}.csv"
                w = cache.get(bucket, bpath, header_small)
                # only the fields we need, so the partitions stay lean
                w.writerow([row.get(c, "") for c in header_small])
    finally:
        cache.close_all()

    # pass 2: aggregate each bucket and hand it to the final writer
    writer, out_fh, _ = _open_out_csv(
        out_path,
        include_source_col,
        include_source_path,
        include_group_count=True,
        id_col_names=None,
        group_by_names=group_by,
        carry_col_names=carried,
        agg_col_names=[n for c, s in agg.items() for n in (f"{c}_{s}", f"{c}_n")],
        include_text=bool(text_cols),
    )

    buckets = sorted(part_dir.glob("bucket_*.csv"))
    ticker = Ticker(on_progress, len(buckets))
    try:
        for bfile in buckets:
            ticker.tick(message="combining the groups (pass 2 of 2)")
            # a bucket is every row whose key hashed here, which with a
            # two-level key is half the file. so the rows are not kept: each
            # group remembers its text pieces, whether its carried columns
            # agree, and running summaries, and the rows go by once.
            # summaries see every row in the group, textless ones included:
            # a person's average score is the average over what they posted,
            # not over what happened to have text. group_count still counts
            # only the rows whose text got joined, and the per-column n makes
            # the difference visible instead of leaving it for someone to
            # discover in an audit. the carried columns have to agree across
            # the rows that contributed text, same as before.
            groups: Dict[Tuple[str, ...], _Group] = {}
            with bfile.open("r", newline="", encoding=encoding) as bf:
                br = csv.DictReader(bf, delimiter=delimiter)
                for row in br:
                    key = tuple(row[g] for g in group_by)
                    group = groups.get(key)
                    if group is None:
                        group = groups[key] = _Group(carried, agg)
                    if agg:
                        group.see_agg(row)
                    if mode == "concat":
                        parts = [row.get(c, "") for c in text_cols if row.get(c, "")]
                        if text_cols and not parts:
                            continue
                        group.pieces.append(joiner.join(parts))
                        if carried:
                            group.see_carried(carried, row)
                    else:
                        wrote = False
                        for col in text_cols:
                            val = row.get(col, "")
                            if val:
                                group.per_col.setdefault(col, []).append(val)
                                wrote = True
                        if carried and wrote:
                            group.see_carried(carried, row)
            # now we write out one row per key (per column, in separate mode)
            for key, group in groups.items():
                text_id = _compose_id(key) or "group"
                kept = [v for n, v in zip(group_by, key, strict=True) if n != "text_id"]
                extra = group.agreed_cells(carried) if carried else []
                stats = group.stat_cells(agg) if agg else []
                if mode == "concat":
                    if not group.pieces:
                        continue
                    body = [joiner.join(group.pieces)] if text_cols else []
                    writer.writerow([text_id, *kept, *extra, *body, len(group.pieces), *stats])
                else:
                    for col in text_cols:
                        vals = group.per_col.get(col, [])
                        if not vals:
                            continue
                        writer.writerow([text_id, *kept, *extra, joiner.join(vals),
                                         len(vals), *stats, col])
            groups.clear()

    except BaseException:
        out_fh.close(ok=False)
        raise
    else:
        out_fh.close(ok=True)
    finally:
        # lastly, clean up after ourselves
        try:
            for p in part_dir.glob("bucket_*.csv"):
                p.unlink(missing_ok=True)
            part_dir.rmdir()
            tmp_base.rmdir()
        except Exception:
            pass

    return tables.handle(out_path)

resolve_analysis_ready

resolve_analysis_ready(
    *,
    csv_path=None,
    txt_dir=None,
    analysis_csv=None,
    gathered_csv=None,
    text_cols=("text",),
    id_cols=None,
    mode="concat",
    group_by=None,
    delimiter=",",
    encoding="utf-8-sig",
    joiner=" ",
    num_buckets=64,
    max_open_bucket_files=64,
    tmp_root=None,
    recursive=True,
    pattern="*.txt",
    id_from="stem",
    include_source_path=False,
    overwrite_existing=False,
    on_progress=None,
    workers=0,
    carry_cols=None,
    verbose=None
)

Accept an analysis-ready table, or gather one -- the analyzers' front door.

Every text analyzer takes its input three ways: a spreadsheet to gather from, a folder of documents to gather from, or a table already gathered. The forty lines that told those apart, announced the gather and called one of the two gatherers with a dozen forwarded settings were copied into eleven modules, and had begun to drift (one forwarded verbose and carry_cols, the others did not). This is that block, once. Settings recording is unaffected: the provenance decorator reads the analyzer's own bound arguments, not this function's.

carry_cols and verbose are forwarded to the gatherers only when given, so the analyzers that never passed them behave exactly as before.

Source code in src\taters\helpers\text_gather.py
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
def resolve_analysis_ready(*, csv_path=None, txt_dir=None, analysis_csv=None,
                           gathered_csv=None, text_cols=("text",), id_cols=None,
                           mode="concat", group_by=None, delimiter=",",
                           encoding="utf-8-sig", joiner=" ", num_buckets=64,
                           max_open_bucket_files=64, tmp_root=None,
                           recursive=True, pattern="*.txt", id_from="stem",
                           include_source_path=False, overwrite_existing=False,
                           on_progress=None, workers=0, carry_cols=None,
                           verbose=None) -> Path:
    """
    Accept an analysis-ready table, or gather one -- the analyzers' front door.

    Every text analyzer takes its input three ways: a spreadsheet to gather
    from, a folder of documents to gather from, or a table already gathered.
    The forty lines that told those apart, announced the gather and called
    one of the two gatherers with a dozen forwarded settings were copied into
    eleven modules, and had begun to drift (one forwarded ``verbose`` and
    ``carry_cols``, the others did not). This is that block, once. Settings
    recording is unaffected: the provenance decorator reads the analyzer's
    own bound arguments, not this function's.

    ``carry_cols`` and ``verbose`` are forwarded to the gatherers only when
    given, so the analyzers that never passed them behave exactly as before.
    """
    # a file comes back as the Path it always was -- every analyzer that
    # still opens it by hand keeps working -- and only a table in a
    # database comes back as a ref
    from .tables import exists, handle, parse_ref

    if analysis_csv is not None:
        analysis_ready = parse_ref(analysis_csv)
        if not exists(analysis_ready):
            raise FileNotFoundError(f"analysis_csv not found: {analysis_ready}")
        return handle(analysis_ready)
    if (csv_path is None) == (txt_dir is None):
        raise ValueError(
            "Provide exactly one of csv_path or txt_dir (or pass analysis_csv).")

    from .progress import announce

    # gathering reads the whole source and writes the analysis-ready table.
    # on a big spreadsheet that's the longest silent stretch in the run, so
    # we say what's happening before it starts.
    announce(on_progress, "reading the input")
    extra = {}
    if verbose is not None:
        extra["verbose"] = verbose
    if csv_path is not None:
        if carry_cols:
            extra["carry_cols"] = list(carry_cols)
        return handle(parse_ref(csv_to_analysis_ready_csv(
            csv_path=csv_path, out_csv=gathered_csv,
            text_cols=list(text_cols),
            id_cols=list(id_cols) if id_cols else None,
            mode=mode, group_by=list(group_by) if group_by else None,
            delimiter=delimiter, encoding=encoding, joiner=joiner,
            num_buckets=num_buckets,
            max_open_bucket_files=max_open_bucket_files, tmp_root=tmp_root,
            overwrite_existing=overwrite_existing, **extra)))
    return handle(parse_ref(txt_folder_to_analysis_ready_csv(
        root_dir=txt_dir, out_csv=gathered_csv, recursive=recursive,
        pattern=pattern, encoding=encoding, id_from=id_from,
        include_source_path=include_source_path,
        overwrite_existing=overwrite_existing, on_progress=on_progress,
        workers=workers, **extra)))

txt_folder_to_analysis_ready_csv

txt_folder_to_analysis_ready_csv(
    *,
    root_dir,
    out_csv=None,
    recursive=False,
    pattern=DOCUMENT_PATTERN,
    encoding="utf-8",
    id_from="stem",
    include_source_path=True,
    overwrite_existing=False,
    verbose=True,
    on_progress=None,
    workers=0
)

Stream a folder of documents into an analysis-ready CSV with predictable, reproducible IDs.

Documents are .txt, .docx, .doc and .pdf -- text is extracted per type by :func:taters.helpers.doc_text.read_document_text (no OCR: a PDF without a machine-readable text layer has zero text). A document that cannot be read -- corrupt, password-protected, a legacy .doc with no way to convert it -- is skipped with a warning naming it, never allowed to take the whole gather down.

For each readable file matching pattern, the emitted row contains: - text_id: the basename (stem), full filename, or relative path (see id_from), and - text: the extracted text. - source_path: optional column with path relative to root_dir.

Parameters:

Name Type Description Default
root_dir PathLike

Folder containing documents.

required
out_csv PathLike | None

Destination CSV. If None, a descriptive default is created next to root_dir (e.g., <folder>_txt_recursive_*.csv).

None
recursive bool

Recurse into subfolders. Default: False.

False
pattern str

Glob(s) for matching files; several can be joined with ;. Default: every document type ("*.txt;*.docx;*.doc;*.pdf").

DOCUMENT_PATTERN
encoding str

Decoding for plain-text files. Default: "utf-8".

'utf-8'
workers int

Parallel reader processes -- PDF and Word parsing is CPU-bound, and a big folder reads several times faster in parallel. 0 (default) means automatic: three-quarters of the logical cores; 1 turns parallelism off. The output file is identical whatever the worker count: files are walked in sorted order and results are written in that same order.

0
id_from str

How to derive text_id: "stem" (basename without extension), "name" (filename), or "path" (relative path).

'stem'
include_source_path bool

If True (default), add a source_path column showing the relative path.

True
overwrite_existing bool

If False (default) and out_csv exists, returns the existing file.

False

Returns:

Type Description
Path

Path to the analysis-ready CSV.

Examples:

>>> txt_folder_to_analysis_ready_csv(root_dir="notes", recursive=True, id_from="path")
Source code in src\taters\helpers\text_gather.py
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
@records_settings(
    # one text per file, so there's no grain to record here
    binding=("root_dir", "recursive", "pattern", "encoding",
             "id_from", "include_source_path"),
    outputs=("out_csv",),
    redo_without_record=True,
    # this step decides what text EXISTS. its settings describe the
    # dataset, so we record them but keep them out of the chain that
    # downstream steps compare -- a model has to be applicable to a corpus
    # assembled differently, which is the whole point of saving one.
    defines_text=True)
def txt_folder_to_analysis_ready_csv(
    *,
    root_dir: PathLike,
    out_csv: PathLike | None = None,
    recursive: bool = False,
    pattern: str = DOCUMENT_PATTERN,
    encoding: str = "utf-8",
    id_from: str = "stem",            # "stem" | "name" | "path"
    include_source_path: bool = True, # writes 'source_path' column
    overwrite_existing: bool = False, # if the file already exists, let's not overwrite by default
    verbose: bool = True,
    on_progress=None,                 # on_progress(done, total, message) per file
    workers: int = 0,                 # parallel readers; 0 = auto, 1 = off
) -> Path:
    """
    Stream a folder of documents into an analysis-ready CSV with predictable,
    reproducible IDs.

    Documents are ``.txt``, ``.docx``, ``.doc`` and ``.pdf`` -- text is
    extracted per type by :func:`taters.helpers.doc_text.read_document_text`
    (no OCR: a PDF without a machine-readable text layer has zero text). A
    document that cannot be read -- corrupt, password-protected, a legacy
    ``.doc`` with no way to convert it -- is skipped with a warning naming it,
    never allowed to take the whole gather down.

    For each readable file matching `pattern`, the emitted row contains:
      - `text_id`: the basename (stem), full filename, or relative path (see
        `id_from`), and
      - `text`: the extracted text.
      - `source_path`: optional column with path relative to `root_dir`.

    Parameters
    ----------
    root_dir
        Folder containing documents.
    out_csv
        Destination CSV. If `None`, a descriptive default is created next to
        `root_dir` (e.g., `<folder>_txt_recursive_*.csv`).
    recursive
        Recurse into subfolders. Default: `False`.
    pattern
        Glob(s) for matching files; several can be joined with ``;``.
        Default: every document type (``"*.txt;*.docx;*.doc;*.pdf"``).
    encoding
        Decoding for plain-text files. Default: `"utf-8"`.
    workers
        Parallel reader processes -- PDF and Word parsing is CPU-bound, and a
        big folder reads several times faster in parallel. ``0`` (default)
        means automatic: three-quarters of the logical cores; ``1`` turns parallelism off.
        The output file is **identical whatever the worker count**: files are
        walked in sorted order and results are written in that same order.
    id_from
        How to derive `text_id`: `"stem"` (basename without extension),
        `"name"` (filename), or `"path"` (relative path).
    include_source_path
        If `True` (default), add a `source_path` column showing the relative path.
    overwrite_existing
        If `False` (default) and `out_csv` exists, returns the existing file.

    Returns
    -------
    Path
        Path to the analysis-ready CSV.

    Examples
    --------
    >>> txt_folder_to_analysis_ready_csv(root_dir="notes", recursive=True, id_from="path")
    """
    from . import tables

    root = _ensure_path(root_dir)
    out_path = tables.parse_ref(out_csv) if out_csv is not None else tables.parse_ref(
        _default_txt_out_path(root, id_from=id_from, recursive=recursive, pattern=pattern))
    if not out_path.is_db:
        out_path.path.parent.mkdir(parents=True, exist_ok=True)

    if not overwrite_existing and tables.exists(out_path):
        if verbose:
            print("File with gathered text already exists; returning existing file.")
        return tables.handle(out_path)

    writer, fh, _ = _open_out_csv(out_path, include_source_col=False, include_source_path=include_source_path)
    try:
        # several globs, but one deduplicated, *sorted* walk: with more than one
        # pattern the concatenation order would otherwise be up to the
        # filesystem, and two runs over the same folder need to write the same
        # file.
        found: set = set()
        for one_pattern in pattern.split(";"):
            one_pattern = one_pattern.strip()
            if one_pattern:
                found.update(root.rglob(one_pattern) if recursive
                             else root.glob(one_pattern))
        files = [p for p in sorted(found) if p.is_file()]
        if id_from not in ("stem", "name", "path"):
            raise ValueError("id_from must be 'stem', 'name', or 'path'")

        # the gather gets to be its own visible phase: the step row reads
        # "gathering N documents" and (on displays that can show it) each
        # document being read gets a sub-bar of its own -- same language as
        # ffmpeg's or Whisper's per-file rows, rather than a step label that
        # looks like it's doing somebody else's work.
        from .progress import FlightReporter

        reporter = FlightReporter(on_progress, len(files),
                                  "gathering documents") \
            if on_progress is not None else None
        for p, (status, payload, relayed) in _pair_paths_with_results(
                files, workers, encoding, reporter):
            if reporter is not None:
                reporter.consumed()
            for message in relayed:
                # warnings raised inside a worker process would otherwise just
                # vanish; they ride back with the result and we re-issue them here.
                warnings.warn(message, stacklevel=2)
            if status == "error":
                # a broken document -- an image wearing a .docx name, a
                # truncated PDF, an unreadable file -- only costs itself: we
                # treat it as having no text, name it in a warning, and never
                # let it take the whole run down.
                warnings.warn(f"{payload} Skipped.", stacklevel=2)
                continue
            text = payload
            if id_from == "stem":
                text_id = p.stem
            elif id_from == "name":
                text_id = p.name
            else:
                text_id = str(p.relative_to(root))
            if len(text) > 500_000:
                # a whole proceedings volume or book as one "document".
                # perfectly legal, but dictionary-style scoring grows
                # super-linearly with length, so one of these can take minutes
                # while its neighbors take milliseconds -- better to say so
                # now, and name it, than to look hung later.
                warnings.warn(
                    f"'{p.name}' extracted {len(text):,} characters -- a very "
                    "large document; some analyses can be slow on it.",
                    stacklevel=2)
            if not text.strip() and p.suffix.lower() != ".txt":
                # a scanned PDF, an image-only Word file: zero text is the
                # documented answer, but if we stayed quiet it'd look like we
                # lost their data.
                warnings.warn(
                    f"'{p.name}' contains no machine-readable text (no OCR "
                    "is attempted); it contributes nothing.", stacklevel=2)
                continue
            if include_source_path:
                writer.writerow([text_id, text, str(p.relative_to(root))])
            else:
                writer.writerow([text_id, text])
    except BaseException:
        fh.close(ok=False)
        raise
    else:
        fh.close(ok=True)

    return tables.handle(out_path)

taters.helpers.reshape

Merge, stack and reshape spreadsheets, without holding any of them in memory.

Four things a person does to a spreadsheet before it is ready to analyze, and that until now meant leaving Taters for R or pandas:

  • :func:merge_spreadsheets -- add the columns of one file onto the rows of another, matching rows by an ID they share (a participant's age and condition onto every one of their trials).
  • :func:stack_spreadsheets -- several files one under another (one per session, per wave, per participant), with a column saying which.
  • :func:wide_to_long -- score_t1, score_t2, score_t3 into one score column plus a column saying which time.
  • :func:long_to_wide -- the other way round.

Every one of them has a planner (:func:plan_merge and friends) that reads the files, writes nothing, and comes back with a report: how many rows match, which don't, what will collide. The wizard reads that report back in words before anything is written, and the writers run the same plan again and refuse the same things, so the API and the app cannot disagree.

Two rules shape everything here. Nothing holds a file's rows: the stacks and the unpivot stream, and the merge and the pivot go through an on-disk sqlite database in a temp folder, the way the n-gram counter already spills. And keys are compared as text, never as floats: a 17-digit ID must never equal its neighbor because they round the same.

Unlike the text gathers, encoding and delimiter default to detect rather than to utf-8-sig and comma. The files this handles are the messy ones -- a cp1252 export from Windows, a semicolon file from a European Excel, a tab-separated "Unicode Text" dump -- and a default that only works on the tidy ones is the wrong default. The output is always utf-8-sig and comma-separated, which every other Taters step reads.

ColumnClash dataclass

ColumnClash(
    name, agrees, disagreements, example, output_names
)

A non-key column present in both files.

MergeReport dataclass

MergeReport(
    left,
    right,
    on,
    right_on,
    key_match,
    relationship,
    many_to_many_keys,
    many_to_many_examples,
    rows_out,
    loose_extra_matches,
    loose_examples,
    clashes,
    columns_out,
    what="file",
    fan_out_examples=(),
)

What a merge would do, before it does it.

refusal

refusal(*, allow_many_to_many=False)

Why the merge must not be written as asked, or None.

Source code in src\taters\helpers\reshape.py
557
558
559
560
561
562
563
564
565
566
567
568
def refusal(self, *, allow_many_to_many: bool = False) -> Optional[str]:
    """Why the merge must not be written as asked, or None."""
    if self.relationship == "many-to-many" and not allow_many_to_many:
        shown = "; ".join(f"{k}: {nl} rows in {self.left.file} × {nr} in "
                          f"{self.right.file} = {nl * nr} combined rows"
                          for k, nl, nr in self.many_to_many_examples)
        return (f"Some {_display(self.on)} values appear more than once in "
                f"both {self.what}s ({shown}; {self.many_to_many_keys:,} like "
                f"this). That usually means one more column is needed to "
                f"tell rows apart -- a session or a time point. Pass "
                f"allow_many_to_many=True to combine every pair anyway.")
    return None

sentences

sentences(how='left')

The report in words, as the wizard reads it back.

Source code in src\taters\helpers\reshape.py
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
def sentences(self, how: str = "left") -> List[str]:
    """The report in words, as the wizard reads it back."""
    left, right, keys = self.left, self.right, _display(self.on)
    out: List[str] = []
    if not left.rows_matched and left.rows > left.blank_keys:
        # nothing matched at all: the formats side by side are the
        # fastest way to see "P001 on one side, 1 on the other"
        out.append(f"No row in {left.file} matches a row in {right.file}. "
                   f"{left.file} keys look like: "
                   f"{', '.join(left.key_samples) or '(none)'}. "
                   f"{right.file} keys look like: "
                   f"{', '.join(right.key_samples) or '(none)'}.")
    elif left.rows_unmatched:
        eg = ", ".join(left.unmatched_examples)
        out.append(f"{left.rows_matched:,} of {left.rows:,} rows in "
                   f"{left.file} match a row in {right.file}; "
                   f"{left.rows_unmatched:,} do not (e.g. {eg}).")
    elif left.rows_matched:
        out.append(f"Every row in {left.file} with a {keys} matches a row "
                   f"in {right.file}.")
    if right.rows_unmatched:
        eg = ", ".join(right.unmatched_examples)
        out.append(f"{right.rows_unmatched:,} of {right.rows:,} rows in "
                   f"{right.file} have no match in {left.file} (e.g. {eg}).")
    for side in (left, right):
        if side.blank_keys:
            out.append(f"{side.blank_keys:,} rows in {side.file} have no "
                       f"{keys}; they can't match anything.")
    if self.relationship == "many-to-many":
        out.append(self.refusal() or "")
    elif self.rows_out.get("inner", 0) > left.rows_matched:
        out.append(f"Some rows in {left.file} match several rows in "
                   f"{right.file} and will repeat once per match: "
                   f"{self.rows_out[how]:,} rows out of {left.rows:,}.")
    for clash in self.clashes:
        both = f"both {self.what}s"
        if clash.agrees:
            out.append(f"{clash.name} is in {both} and agrees on every "
                       f"matched row, so it's kept once.")
        else:
            k, a, b = clash.example or ("", "", "")
            out.append(f"{clash.name} is in {both} and differs on "
                       f"{clash.disagreements:,} matched rows (e.g. {k}: "
                       f"{a!r} vs {b!r}); both are kept, as "
                       f"{' and '.join(clash.output_names)}.")
    if self.loose_extra_matches:
        l, r = self.loose_examples[0] if self.loose_examples else ("", "")
        out.append(f"Some keys only differ in how they're written ({l} in "
                   f"{left.file} is {r} in {right.file}); matching those "
                   f"too would add {self.loose_extra_matches:,} rows.")
    for side in (left, right):
        if side.loose_collisions:
            a, b = side.loose_collision_examples[0]
            out.append(f"{side.file} has keys that differ only in case or "
                       f"spacing ({a}, {b}), so those differences matter "
                       f"there.")
        if side.mangled_ids:
            out.append(f"{side.mangled_ids:,} keys in {side.file} look like "
                       f"1.23457E+11, which Excel writes after rounding a "
                       f"long number; those can't be matched reliably. "
                       f"Re-export that column as text.")
    return [s for s in out if s]

SideSummary dataclass

SideSummary(
    file,
    encoding,
    delimiter,
    rows,
    keys,
    renamed_headers,
    ragged_short,
    ragged_long,
    blank_keys,
    distinct_keys,
    duplicated_keys,
    rows_matched,
    rows_unmatched,
    unmatched_keys,
    unmatched_examples,
    loose_collisions,
    loose_collision_examples,
    mangled_ids,
    key_samples,
)

One file's half of a merge report.

TableFormat dataclass

TableFormat(path, encoding, delimiter, columns, renamed=())

How a spreadsheet has to be read, worked out from the file itself.

columns are the names as this module will call them: stripped of surrounding spaces and any stray BOM, a blank header filled in as column_<n>, a duplicate made unique as name_2. renamed says which ones changed, so a screen can mention it.

notes

notes()

What a person should know about how the file was read, or [].

Source code in src\taters\helpers\reshape.py
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
def notes(self) -> List[str]:
    """What a person should know about how the file was read, or []."""
    out = []
    if self.encoding == "utf-16":
        out.append(f"{self.name} is a UTF-16 file (Excel's \"Unicode Text\"), "
                   f"and was read as one.")
    elif self.encoding in ("cp1252", "latin-1"):
        out.append(f"{self.name} isn't UTF-8, so it was read as Windows "
                   f"({self.encoding}) text. Accented letters should come "
                   f"through fine.")
    if self.delimiter != ",":
        word = {"\t": "tab", ";": "semicolon", "|": "pipe"}[self.delimiter]
        out.append(f"{self.name} is {word}-separated.")
    for was, now in self.renamed:
        if was:
            out.append(f"{self.name} has more than one column called "
                       f"{was}; the next one is {now}.")
        else:
            out.append(f"A column in {self.name} had no name; it's called "
                       f"{now}.")
    return out

detect_format

detect_format(path, *, encoding=None, delimiter=None)

Work out how to read a spreadsheet: its encoding, separator and columns.

Parameters:

Name Type Description Default
path str or Path

The spreadsheet.

required
encoding str

Use this instead of detecting. Detection checks for a BOM, then whether the whole file is UTF-8, then falls back to cp1252 and finally latin-1 -- which reads anything, so a file is never refused for its encoding.

None
delimiter str

Use this instead of sniffing. Sniffing is done on decoded text, so a UTF-16 file sniffs correctly.

None

Returns:

Type Description
TableFormat

Raises:

Type Description
FileNotFoundError

No such file.

ValueError

The file is empty, has no header row, or does not look like a CSV at all (an .xlsx or .docx is a zip, and its first bytes decode as text full of control characters).

Source code in src\taters\helpers\reshape.py
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
def detect_format(path: PathLike, *, encoding: Optional[str] = None,
                  delimiter: Optional[str] = None) -> TableFormat:
    """
    Work out how to read a spreadsheet: its encoding, separator and columns.

    Parameters
    ----------
    path : str or Path
        The spreadsheet.
    encoding : str, optional
        Use this instead of detecting. Detection checks for a BOM, then
        whether the whole file is UTF-8, then falls back to cp1252 and
        finally latin-1 -- which reads anything, so a file is never refused
        for its encoding.
    delimiter : str, optional
        Use this instead of sniffing. Sniffing is done on decoded text, so a
        UTF-16 file sniffs correctly.

    Returns
    -------
    TableFormat

    Raises
    ------
    FileNotFoundError
        No such file.
    ValueError
        The file is empty, has no header row, or does not look like a CSV
        at all (an .xlsx or .docx is a zip, and its first bytes decode as
        text full of control characters).
    """
    path = Path(path)
    if not path.is_file():
        raise FileNotFoundError(f"{path} does not exist")
    if path.stat().st_size == 0:
        raise ValueError(f"{path.name} is empty")
    encoding = encoding or _detect_encoding(path)
    delimiter = delimiter or _sniff(path, encoding)
    try:
        with path.open("r", newline="", encoding=encoding) as fh:
            header = next(csv.reader(fh, delimiter=delimiter), None)
    except UnicodeDecodeError:
        raise ValueError(
            f"{path.name} isn't readable as {encoding}; leave the encoding "
            f"unset and Taters will work it out.") from None
    if not header or not any(str(c).strip() for c in header):
        raise ValueError(f"{path.name} has no header row to read columns from")
    if any(ch < " " and ch not in "\t\r\n" for cell in header for ch in str(cell)):
        raise ValueError(
            f"{path.name} looks like an Excel or Word file, not a CSV. In "
            f"Excel: File, Save As, CSV UTF-8.")
    names, renamed = _unique_names(header)
    return TableFormat(path=path, encoding=encoding, delimiter=delimiter,
                       columns=tuple(names), renamed=tuple(renamed))

long_to_wide

long_to_wide(
    *,
    csv_path,
    id_cols,
    names_from,
    values_from,
    out_csv=None,
    overwrite_existing=False,
    carry_cols=None,
    on_duplicate="refuse",
    max_columns=1000,
    key_match="trim",
    delimiter=None,
    encoding=None,
    tmp_root=None,
    verbose=True,
    on_progress=None
)

Turn rows into columns: one row per key, one column per name.

Parameters:

Name Type Description Default
csv_path str or Path

The long spreadsheet.

required
id_cols sequence of str

The column(s) that say which rows belong together -- one output row per distinct combination, in order of first appearance.

required
names_from str

The column whose values become column names: a condition, a time point, a question number.

required
values_from str or sequence of str

The column holding the values that go into them. Several columns give <column>_<name> for each.

required
out_csv str or Path

Where to write. Defaults to <stem>_wide.csv beside the input.

None
overwrite_existing bool

Rebuild the output if it is already there.

False
carry_cols sequence of str

Other columns to bring along. Each has to hold one value per key (a blank defers to the other rows); one that varies is refused, because in a pivot that means the key is missing a column.

None
on_duplicate ('refuse', 'first', 'last', 'mean')

What to do when one key has the same name on several rows. Refused by default, with examples, since that nearly always means a missing key column. mean needs every repeated value to be a number.

"refuse"
max_columns int

Refuse to make more generated columns than this. An ID or timestamp picked as names_from is the usual way past it.

1000
key_match ('exact', 'trim', 'loose')

How the key values are compared; see :func:normalize_key.

"exact"
delimiter str

Detected when omitted.

None
encoding str

Detected when omitted.

None
tmp_root str or Path

Where the working database goes meanwhile; removed afterwards.

None
verbose bool

Print what was done.

True
on_progress callable

Progress callback.

None

Returns:

Type Description
Path

out_csv.

Notes

Generated columns are bare names (A, B) when there is one value column and none would collide with an existing column; otherwise every one is <values_from>_<name>, so the naming is consistent rather than piecemeal. Names that are all numbers come out in numeric order. A row with a blank key or a blank name is left out and counted. Missing combinations are blank.

Source code in src\taters\helpers\reshape.py
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
@records_settings(
    binding=("csv_path", "names_from", "values_from", "carry_cols",
             "on_duplicate", "max_columns", "key_match", "delimiter", "encoding"),
    grain=("id_cols",),
    outputs=("out_csv",),
    redo_without_record=True)
def long_to_wide(
    *,
    csv_path: PathLike,
    id_cols: Sequence[str],
    names_from: str,
    values_from: Union[str, Sequence[str]],
    out_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    carry_cols: Optional[Sequence[str]] = None,
    on_duplicate: str = "refuse",
    max_columns: int = 1000,
    key_match: str = "trim",
    delimiter: Optional[str] = None,
    encoding: Optional[str] = None,
    tmp_root: Optional[PathLike] = None,
    verbose: bool = True,
    on_progress: Progress = None,
) -> Path:
    """
    Turn rows into columns: one row per key, one column per name.

    Parameters
    ----------
    csv_path : str or Path
        The long spreadsheet.
    id_cols : sequence of str
        The column(s) that say which rows belong together -- one output row
        per distinct combination, in order of first appearance.
    names_from : str
        The column whose values become column names: a condition, a time
        point, a question number.
    values_from : str or sequence of str
        The column holding the values that go into them. Several columns
        give ``<column>_<name>`` for each.
    out_csv : str or Path, optional
        Where to write. Defaults to ``<stem>_wide.csv`` beside the input.
    overwrite_existing : bool, default False
        Rebuild the output if it is already there.
    carry_cols : sequence of str, optional
        Other columns to bring along. Each has to hold one value per key (a
        blank defers to the other rows); one that varies is refused, because
        in a pivot that means the key is missing a column.
    on_duplicate : {"refuse", "first", "last", "mean"}
        What to do when one key has the same name on several rows. Refused
        by default, with examples, since that nearly always means a missing
        key column. ``mean`` needs every repeated value to be a number.
    max_columns : int, default 1000
        Refuse to make more generated columns than this. An ID or timestamp
        picked as ``names_from`` is the usual way past it.
    key_match : {"exact", "trim", "loose"}
        How the key values are compared; see :func:`normalize_key`.
    delimiter, encoding : str, optional
        Detected when omitted.
    tmp_root : str or Path, optional
        Where the working database goes meanwhile; removed afterwards.
    verbose : bool, default True
        Print what was done.
    on_progress : callable, optional
        Progress callback.

    Returns
    -------
    pathlib.Path
        ``out_csv``.

    Notes
    -----
    Generated columns are bare names (``A``, ``B``) when there is one value
    column and none would collide with an existing column; otherwise every
    one is ``<values_from>_<name>``, so the naming is consistent rather than
    piecemeal. Names that are all numbers come out in numeric order. A row
    with a blank key or a blank name is left out and counted. Missing
    combinations are blank.
    """
    job = _Pivot(csv_path, id_cols=id_cols, names_from=names_from,
                 values_from=values_from, carry_cols=carry_cols,
                 on_duplicate=on_duplicate, max_columns=max_columns,
                 key_match=key_match, delimiter=delimiter, encoding=encoding,
                 tmp_root=tmp_root)
    src = job.fmt.path
    out = Path(out_csv) if out_csv is not None else \
        src.with_name(f"{src.stem}_wide.csv")
    _check_output(out, [src])
    if out.is_file() and not overwrite_existing:
        if verbose:
            print(f"[reshape] {out} exists; leaving it (overwrite_existing=False)")
        return out
    with job:
        announce(on_progress, f"reading {src.name}")
        job.load()
        report = job.report()
        why = report.refusal()
        if why:
            raise ValueError(why)
        announce(on_progress, f"writing {out.name}")
        ticker = Ticker(on_progress, total=report.rows_out)
        written = job.write(out, on_rows=_counting(ticker))
    if written != report.rows_out:
        raise RuntimeError(f"long_to_wide wrote {written} rows but planned "
                           f"{report.rows_out}")
    if verbose:
        for line in report.sentences():
            print(f"[reshape] {line}")
        print(f"[reshape] {src.name} -> {out} ({written:,} rows)")
    return out

merge_spreadsheets

merge_spreadsheets(
    *,
    left_csv,
    right_csv,
    on,
    right_on=None,
    out_csv=None,
    overwrite_existing=False,
    how="left",
    key_match="trim",
    right_cols=None,
    allow_many_to_many=False,
    left_delimiter=None,
    right_delimiter=None,
    left_encoding=None,
    right_encoding=None,
    tmp_root=None,
    verbose=True,
    on_progress=None
)

Add the columns of one spreadsheet onto the rows of another, matched by key.

Parameters:

Name Type Description Default
left_csv str or Path

The file whose rows you are keeping -- the trials, the posts, the utterances. Its rows come out in their own order.

required
right_csv str or Path

The file with the columns to add -- one row per participant, say.

required
on str or sequence of str

The column(s) that identify the same thing in both files. Several names make a composite key (participant and session together).

required
right_on str or sequence of str

The same key under different names in right_csv, paired with on in order. Omit when the names match.

None
out_csv str or Path

Where to write. Defaults to <left>_with_<right>.csv beside the left file. Never one of the inputs.

None
overwrite_existing bool

Rebuild the output if it is already there; otherwise it is returned as it is.

False
how ('left', 'inner', 'outer')

left keeps every left row, with blank added columns where nothing matched. inner keeps only the matched rows. outer also appends the right rows nothing matched, at the end.

"left"
key_match ('exact', 'trim', 'loose')

How keys are compared; see :func:normalize_key. trim by default. loose is what makes 007 match 7 and P01 match p01, and is worth turning on after :func:plan_merge has shown what it would add.

"exact"
right_cols sequence of str

Only these right columns are brought over. All of them by default.

None
allow_many_to_many bool

A key that repeats in both files multiplies rows, and is almost always a sign that one more key column is needed. Refused unless this is set.

False
left_delimiter str

The separators. Sniffed per file when omitted.

None
right_delimiter str

The separators. Sniffed per file when omitted.

None
left_encoding str

The encodings. Detected per file when omitted: UTF-8 (with or without a BOM), UTF-16, then cp1252, then latin-1.

None
right_encoding str

The encodings. Detected per file when omitted: UTF-8 (with or without a BOM), UTF-16, then cp1252, then latin-1.

None
tmp_root str or Path

Where the working database goes while the merge runs. It is about the size of the right file and is removed afterwards.

None
verbose bool

Print the plan's sentences and the row count.

True
on_progress callable

Progress callback.

None

Returns:

Type Description
Path

out_csv.

Notes

A non-key column present in both files is kept once when the two agree on every matched row (blank on either side counts as agreement, and the left's blanks are filled from the right); otherwise the right copy is renamed <right stem>__<column>. A blank key never matches anything. Rows with no match are counted and the first few are named, so a key typed as P001 on one side and 1 on the other is found before the analysis, not after.

Source code in src\taters\helpers\reshape.py
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
@records_settings(
    # which files were read and how their rows were paired. recorded for
    # diagnosis, never compared -- and because the inputs are bound, a
    # feature table merged with its metadata keeps the chain of whatever
    # measured it.
    binding=("left_csv", "right_csv", "on", "right_on", "right_cols",
             "key_match", "allow_many_to_many", "left_delimiter",
             "right_delimiter", "left_encoding", "right_encoding"),
    # which rows survive is what a row *is*, not how anything was measured
    grain=("how",),
    outputs=("out_csv",),
    redo_without_record=True)
def merge_spreadsheets(
    *,
    left_csv: PathLike,
    right_csv: PathLike,
    on: Union[str, Sequence[str]],
    right_on: Union[str, Sequence[str], None] = None,
    out_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    how: str = "left",
    key_match: str = "trim",
    right_cols: Optional[Sequence[str]] = None,
    allow_many_to_many: bool = False,
    left_delimiter: Optional[str] = None,
    right_delimiter: Optional[str] = None,
    left_encoding: Optional[str] = None,
    right_encoding: Optional[str] = None,
    tmp_root: Optional[PathLike] = None,
    verbose: bool = True,
    on_progress: Progress = None,
) -> Path:
    """
    Add the columns of one spreadsheet onto the rows of another, matched by key.

    Parameters
    ----------
    left_csv : str or Path
        The file whose rows you are keeping -- the trials, the posts, the
        utterances. Its rows come out in their own order.
    right_csv : str or Path
        The file with the columns to add -- one row per participant, say.
    on : str or sequence of str
        The column(s) that identify the same thing in both files. Several
        names make a composite key (participant and session together).
    right_on : str or sequence of str, optional
        The same key under different names in ``right_csv``, paired with
        ``on`` in order. Omit when the names match.
    out_csv : str or Path, optional
        Where to write. Defaults to ``<left>_with_<right>.csv`` beside the
        left file. Never one of the inputs.
    overwrite_existing : bool, default False
        Rebuild the output if it is already there; otherwise it is returned
        as it is.
    how : {"left", "inner", "outer"}
        ``left`` keeps every left row, with blank added columns where nothing
        matched. ``inner`` keeps only the matched rows. ``outer`` also
        appends the right rows nothing matched, at the end.
    key_match : {"exact", "trim", "loose"}
        How keys are compared; see :func:`normalize_key`. ``trim`` by default.
        ``loose`` is what makes ``007`` match ``7`` and ``P01`` match ``p01``,
        and is worth turning on after :func:`plan_merge` has shown what it
        would add.
    right_cols : sequence of str, optional
        Only these right columns are brought over. All of them by default.
    allow_many_to_many : bool, default False
        A key that repeats in *both* files multiplies rows, and is almost
        always a sign that one more key column is needed. Refused unless
        this is set.
    left_delimiter, right_delimiter : str, optional
        The separators. Sniffed per file when omitted.
    left_encoding, right_encoding : str, optional
        The encodings. Detected per file when omitted: UTF-8 (with or
        without a BOM), UTF-16, then cp1252, then latin-1.
    tmp_root : str or Path, optional
        Where the working database goes while the merge runs. It is about
        the size of the right file and is removed afterwards.
    verbose : bool, default True
        Print the plan's sentences and the row count.
    on_progress : callable, optional
        Progress callback.

    Returns
    -------
    pathlib.Path
        ``out_csv``.

    Notes
    -----
    A non-key column present in both files is kept once when the two agree
    on every matched row (blank on either side counts as agreement, and the
    left's blanks are filled from the right); otherwise the right copy is
    renamed ``<right stem>__<column>``. A blank key never matches anything.
    Rows with no match are counted and the first few are named, so a key
    typed as ``P001`` on one side and ``1`` on the other is found before the
    analysis, not after.
    """
    if how not in HOW:
        raise ValueError(f"how must be one of {HOW}, not {how!r}")
    engine = _merge_engine(
        left_csv, right_csv, on=on, right_on=right_on, right_cols=right_cols,
        key_match=key_match, left_delimiter=left_delimiter,
        right_delimiter=right_delimiter, left_encoding=left_encoding,
        right_encoding=right_encoding, tmp_root=tmp_root)
    left, right = engine.left, engine.right
    out = Path(out_csv) if out_csv is not None else \
        left.path.with_name(f"{left.path.stem}_with_{right.path.stem}.csv")
    _check_output(out, [left.path, right.path])
    if out.is_file() and not overwrite_existing:
        if verbose:
            print(f"[reshape] {out} exists; leaving it (overwrite_existing=False)")
        return out
    with engine:
        announce(on_progress, f"reading {right.name}")
        engine.load_right(on_rows=_counting(Ticker(on_progress)))
        announce(on_progress, f"matching {left.name}")
        engine.scan_left(on_rows=_counting(Ticker(on_progress)))
        report = engine.report()
        why = report.refusal(allow_many_to_many=allow_many_to_many)
        if why:
            raise ValueError(why)
        if verbose:
            for line in report.sentences(how):
                print(f"[reshape] {line}")
            if not report.matched:
                print("[reshape] WARNING: nothing matched; check that the "
                      "keys are written the same way in both files")
        announce(on_progress, f"writing {out.name}")
        ticker = Ticker(on_progress, total=report.left.rows)
        written = engine.write(out, how, on_rows=_counting(ticker))
    expected = report.rows_out[how]
    if written != expected:          # the plan and the write share one store
        raise RuntimeError(f"merge wrote {written} rows but planned {expected}")
    if verbose:
        print(f"[reshape] {left.name} + {right.name} -> {out} ({written:,} rows, "
              f"{how})")
    return out

normalize_key

normalize_key(value, mode='trim')

One key cell as it is compared.

Parameters:

Name Type Description Default
value any

The cell. None reads as blank.

required
mode ('exact', 'trim', 'loose')

exact compares the text as it stands. trim drops surrounding whitespace, including the non-breaking kind, because invisible padding is never a meaningful difference. loose also drops zero-width characters, collapses inner whitespace to one space, normalizes Unicode to NFC, ignores case, and writes a plain number one way -- so 007, 7, 7.0 and +7 are one key. Never a float: 1e3 is not a number here, and a 17-digit ID cannot collide with its neighbor.

"exact"

Returns:

Type Description
str
Source code in src\taters\helpers\reshape.py
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
def normalize_key(value: Any, mode: str = "trim") -> str:
    """
    One key cell as it is compared.

    Parameters
    ----------
    value : any
        The cell. ``None`` reads as blank.
    mode : {"exact", "trim", "loose"}
        ``exact`` compares the text as it stands. ``trim`` drops surrounding
        whitespace, including the non-breaking kind, because invisible
        padding is never a meaningful difference. ``loose`` also drops
        zero-width characters, collapses inner whitespace to one space,
        normalizes Unicode to NFC, ignores case, and writes a plain number
        one way -- so ``007``, ``7``, ``7.0`` and ``+7`` are one key. Never
        a float: ``1e3`` is not a number here, and a 17-digit ID cannot
        collide with its neighbor.

    Returns
    -------
    str
    """
    if mode not in KEY_MATCH:
        raise ValueError(f"key_match must be one of {KEY_MATCH}, not {mode!r}")
    text = "" if value is None else str(value)
    if mode == "exact":
        return text
    text = text.strip()
    if mode == "trim":
        return text
    text = _INVISIBLE.sub("", text)
    text = _INNER_SPACE.sub(" ", text).strip()
    text = unicodedata.normalize("NFC", text).casefold()
    number = _canonical_number(text)
    return number if number is not None else text

plan_long_to_wide

plan_long_to_wide(
    *,
    csv_path,
    id_cols,
    names_from,
    values_from,
    carry_cols=None,
    on_duplicate="refuse",
    max_columns=1000,
    key_match="trim",
    delimiter=None,
    encoding=None,
    tmp_root=None,
    on_progress=None
)

What :func:long_to_wide would do. Reads the file once.

Source code in src\taters\helpers\reshape.py
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
def plan_long_to_wide(*, csv_path: PathLike, id_cols: Sequence[str],
                      names_from: str, values_from: Union[str, Sequence[str]],
                      carry_cols: Optional[Sequence[str]] = None,
                      on_duplicate: str = "refuse", max_columns: int = 1000,
                      key_match: str = "trim", delimiter: Optional[str] = None,
                      encoding: Optional[str] = None,
                      tmp_root: Optional[PathLike] = None,
                      on_progress: Progress = None) -> PivotReport:
    """What :func:`long_to_wide` would do. Reads the file once."""
    job = _Pivot(csv_path, id_cols=id_cols, names_from=names_from,
                 values_from=values_from, carry_cols=carry_cols,
                 on_duplicate=on_duplicate, max_columns=max_columns,
                 key_match=key_match, delimiter=delimiter, encoding=encoding,
                 tmp_root=tmp_root)
    with job:
        announce(on_progress, f"reading {job.fmt.name}")
        job.load(on_rows=_counting(Ticker(on_progress)))
        return job.report()

plan_merge

plan_merge(
    *,
    left_csv,
    right_csv,
    on,
    right_on=None,
    key_match="trim",
    right_cols=None,
    left_delimiter=None,
    right_delimiter=None,
    left_encoding=None,
    right_encoding=None,
    tmp_root=None,
    on_progress=None
)

What :func:merge_spreadsheets would do, without writing anything.

Same parameters; see there. Reads both files once each.

Source code in src\taters\helpers\reshape.py
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
def plan_merge(*, left_csv: PathLike, right_csv: PathLike,
               on: Union[str, Sequence[str]],
               right_on: Union[str, Sequence[str], None] = None,
               key_match: str = "trim",
               right_cols: Optional[Sequence[str]] = None,
               left_delimiter: Optional[str] = None,
               right_delimiter: Optional[str] = None,
               left_encoding: Optional[str] = None,
               right_encoding: Optional[str] = None,
               tmp_root: Optional[PathLike] = None,
               on_progress: Progress = None) -> MergeReport:
    """
    What :func:`merge_spreadsheets` would do, without writing anything.

    Same parameters; see there. Reads both files once each.
    """
    engine = _merge_engine(
        left_csv, right_csv, on=on, right_on=right_on, right_cols=right_cols,
        key_match=key_match, left_delimiter=left_delimiter,
        right_delimiter=right_delimiter, left_encoding=left_encoding,
        right_encoding=right_encoding, tmp_root=tmp_root)
    with engine:
        announce(on_progress, f"reading {engine.right.name}")
        engine.load_right(on_rows=_counting(Ticker(on_progress)))
        announce(on_progress, f"matching {engine.left.name}")
        engine.scan_left(on_rows=_counting(Ticker(on_progress)))
        return engine.report()

plan_stack

plan_stack(
    *,
    csv_paths,
    source_col="source",
    match_columns="exact",
    delimiter=None,
    encoding=None,
    count_rows=True,
    on_progress=None
)

What :func:stack_spreadsheets would do. Reads each file once.

count_rows=False skips the pass and reports zero rows per file: the column plan alone, from the headers, for a screen that wants to settle how columns match before it reads anything.

Source code in src\taters\helpers\reshape.py
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
def plan_stack(*, csv_paths: Sequence[PathLike], source_col: str = "source",
               match_columns: str = "exact", delimiter: Optional[str] = None,
               encoding: Optional[str] = None, count_rows: bool = True,
               on_progress: Progress = None) -> StackReport:
    """
    What :func:`stack_spreadsheets` would do. Reads each file once.

    ``count_rows=False`` skips the pass and reports zero rows per file: the
    column plan alone, from the headers, for a screen that wants to settle
    how columns match before it reads anything.
    """
    stack = _Stack(csv_paths, source_col=source_col, match_columns=match_columns,
                   delimiter=delimiter, encoding=encoding)
    if count_rows:
        announce(on_progress, "reading the files")
        ticker = Ticker(on_progress)
        for reader in stack.readers:
            reader.on_rows = _counting(ticker)
        stack.count()
    return stack.report()

plan_wide_to_long

plan_wide_to_long(
    *,
    csv_path,
    value_cols,
    id_cols=None,
    names_to="variable",
    values_to="value",
    strip_prefix="",
    drop_blank=False,
    delimiter=None,
    encoding=None,
    on_progress=None
)

What :func:wide_to_long would do. Reads the file once.

Source code in src\taters\helpers\reshape.py
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
def plan_wide_to_long(*, csv_path: PathLike, value_cols: Sequence[str],
                      id_cols: Optional[Sequence[str]] = None,
                      names_to: str = "variable", values_to: str = "value",
                      strip_prefix: str = "", drop_blank: bool = False,
                      delimiter: Optional[str] = None,
                      encoding: Optional[str] = None,
                      on_progress: Progress = None) -> UnpivotReport:
    """What :func:`wide_to_long` would do. Reads the file once."""
    job = _Unpivot(csv_path, value_cols=value_cols, id_cols=id_cols,
                   names_to=names_to, values_to=values_to,
                   strip_prefix=strip_prefix, drop_blank=drop_blank,
                   delimiter=delimiter, encoding=encoding)
    announce(on_progress, f"reading {job.fmt.name}")
    job.reader.on_rows = _counting(Ticker(on_progress))
    job.scan()
    return job.report()

stack_spreadsheets

stack_spreadsheets(
    *,
    csv_paths,
    out_csv=None,
    overwrite_existing=False,
    source_col="source",
    match_columns="exact",
    delimiter=None,
    encoding=None,
    verbose=True,
    on_progress=None
)

Put the rows of several spreadsheets one under another.

Parameters:

Name Type Description Default
csv_paths sequence of str or Path

The files, in the order their rows should come out. At least two. A file listed twice is stacked once.

required
out_csv str or Path

Where to write. Defaults to <first stem>_stacked.csv beside the first file.

None
overwrite_existing bool

Rebuild the output if it is already there.

False
source_col str

A first column naming the file each row came from (the file's stem, or its path under the files' common folder when stems repeat). An empty string leaves it out. If the name is taken, source_file.

"source"
match_columns ('exact', 'loose')

Whether Age and age are the same column. Surrounding spaces never count either way.

"exact"
delimiter str

Applied to every file; detected per file when omitted.

None
encoding str

Applied to every file; detected per file when omitted.

None
verbose bool

Print what was stacked.

True
on_progress callable

Progress callback.

None

Returns:

Type Description
Path

out_csv.

Notes

Columns are the union, in order of first appearance, and a cell a file does not have is blank. A file sharing no column with the others is stacked anyway, and the plan says so: that is usually a merge wanted by mistake.

Source code in src\taters\helpers\reshape.py
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
@records_settings(
    binding=("csv_paths", "match_columns", "delimiter", "encoding"),
    grain=("source_col",),
    outputs=("out_csv",),
    redo_without_record=True)
def stack_spreadsheets(
    *,
    csv_paths: Sequence[PathLike],
    out_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    source_col: str = "source",
    match_columns: str = "exact",
    delimiter: Optional[str] = None,
    encoding: Optional[str] = None,
    verbose: bool = True,
    on_progress: Progress = None,
) -> Path:
    """
    Put the rows of several spreadsheets one under another.

    Parameters
    ----------
    csv_paths : sequence of str or Path
        The files, in the order their rows should come out. At least two.
        A file listed twice is stacked once.
    out_csv : str or Path, optional
        Where to write. Defaults to ``<first stem>_stacked.csv`` beside the
        first file.
    overwrite_existing : bool, default False
        Rebuild the output if it is already there.
    source_col : str, default "source"
        A first column naming the file each row came from (the file's stem,
        or its path under the files' common folder when stems repeat). An
        empty string leaves it out. If the name is taken, ``source_file``.
    match_columns : {"exact", "loose"}
        Whether ``Age`` and ``age`` are the same column. Surrounding spaces
        never count either way.
    delimiter, encoding : str, optional
        Applied to every file; detected per file when omitted.
    verbose : bool, default True
        Print what was stacked.
    on_progress : callable, optional
        Progress callback.

    Returns
    -------
    pathlib.Path
        ``out_csv``.

    Notes
    -----
    Columns are the union, in order of first appearance, and a cell a file
    does not have is blank. A file sharing no column with the others is
    stacked anyway, and the plan says so: that is usually a merge wanted
    by mistake.
    """
    stack = _Stack(csv_paths, source_col=source_col, match_columns=match_columns,
                   delimiter=delimiter, encoding=encoding)
    first = stack.formats[0].path
    out = Path(out_csv) if out_csv is not None else \
        first.with_name(f"{first.stem}_stacked.csv")
    _check_output(out, [f.path for f in stack.formats])
    if out.is_file() and not overwrite_existing:
        if verbose:
            print(f"[reshape] {out} exists; leaving it (overwrite_existing=False)")
        return out
    header = ([stack.source_col] if stack.source_col else []) + stack.columns
    width = len(stack.columns)
    ticker = Ticker(on_progress, total=len(stack.formats))
    written = 0
    with atomic_write(out, "w", newline="", encoding=OUTPUT_ENCODING) as fh:
        w = csv.writer(fh)
        w.writerow(header)
        for fmt, reader, label in zip(stack.formats, stack.readers, stack.labels):
            places = [stack.place[stack.canon(c)] for c in fmt.columns]
            for row in reader:
                line = [""] * width
                for value, at in zip(row, places):
                    line[at] = value
                w.writerow(([label] if stack.source_col else []) + line)
                written += 1
            ticker.tick(1, message=fmt.name)
    if verbose:
        for line in stack.report().sentences():
            print(f"[reshape] {line}")
        print(f"[reshape] stacked {len(stack.formats)} files -> {out} "
              f"({written:,} rows)")
    return out

wide_to_long

wide_to_long(
    *,
    csv_path,
    value_cols,
    id_cols=None,
    out_csv=None,
    overwrite_existing=False,
    names_to="variable",
    values_to="value",
    strip_prefix="",
    drop_blank=False,
    delimiter=None,
    encoding=None,
    verbose=True,
    on_progress=None
)

Turn several columns into rows: one row per (original row, column).

Parameters:

Name Type Description Default
csv_path str or Path

The wide spreadsheet.

required
value_cols sequence of str

The columns that hold the same kind of measurement at different times or in different conditions -- score_t1, score_t2, score_t3. Each becomes its own row. At least two.

required
id_cols sequence of str

The columns that ride along unchanged on every one of those rows. By default every column not in value_cols. If none are left, a row column numbers the original rows so they can be traced.

None
out_csv str or Path

Where to write. Defaults to <stem>_long.csv beside the input.

None
overwrite_existing bool

Rebuild the output if it is already there.

False
names_to str

The new column saying which of value_cols a row came from.

"variable"
values_to str

The new column holding the value.

"value"
strip_prefix str

Removed from the front of every value column's name in names_to (score_t1 becomes 1). Every value column has to start with it.

""
drop_blank bool

Leave out the rows whose value is empty. Kept by default, so a balanced design stays visibly balanced.

False
delimiter str

Detected when omitted.

None
encoding str

Detected when omitted.

None
verbose bool

Print what was done.

True
on_progress callable

Progress callback.

None

Returns:

Type Description
Path

out_csv.

Notes

Values are copied as they are. When some value columns hold numbers and others hold text, the plan says so, because the one value column will then read as text everywhere.

Source code in src\taters\helpers\reshape.py
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
@records_settings(
    binding=("csv_path", "value_cols", "id_cols", "strip_prefix", "drop_blank",
             "delimiter", "encoding"),
    grain=("names_to", "values_to"),
    outputs=("out_csv",),
    redo_without_record=True)
def wide_to_long(
    *,
    csv_path: PathLike,
    value_cols: Sequence[str],
    id_cols: Optional[Sequence[str]] = None,
    out_csv: Optional[PathLike] = None,
    overwrite_existing: bool = False,
    names_to: str = "variable",
    values_to: str = "value",
    strip_prefix: str = "",
    drop_blank: bool = False,
    delimiter: Optional[str] = None,
    encoding: Optional[str] = None,
    verbose: bool = True,
    on_progress: Progress = None,
) -> Path:
    """
    Turn several columns into rows: one row per (original row, column).

    Parameters
    ----------
    csv_path : str or Path
        The wide spreadsheet.
    value_cols : sequence of str
        The columns that hold the same kind of measurement at different
        times or in different conditions -- ``score_t1, score_t2,
        score_t3``. Each becomes its own row. At least two.
    id_cols : sequence of str, optional
        The columns that ride along unchanged on every one of those rows.
        By default every column not in ``value_cols``. If none are left, a
        ``row`` column numbers the original rows so they can be traced.
    out_csv : str or Path, optional
        Where to write. Defaults to ``<stem>_long.csv`` beside the input.
    overwrite_existing : bool, default False
        Rebuild the output if it is already there.
    names_to : str, default "variable"
        The new column saying which of ``value_cols`` a row came from.
    values_to : str, default "value"
        The new column holding the value.
    strip_prefix : str, default ""
        Removed from the front of every value column's name in ``names_to``
        (``score_t1`` becomes ``1``). Every value column has to start with
        it.
    drop_blank : bool, default False
        Leave out the rows whose value is empty. Kept by default, so a
        balanced design stays visibly balanced.
    delimiter, encoding : str, optional
        Detected when omitted.
    verbose : bool, default True
        Print what was done.
    on_progress : callable, optional
        Progress callback.

    Returns
    -------
    pathlib.Path
        ``out_csv``.

    Notes
    -----
    Values are copied as they are. When some value columns hold numbers and
    others hold text, the plan says so, because the one value column will
    then read as text everywhere.
    """
    job = _Unpivot(csv_path, value_cols=value_cols, id_cols=id_cols,
                   names_to=names_to, values_to=values_to,
                   strip_prefix=strip_prefix, drop_blank=drop_blank,
                   delimiter=delimiter, encoding=encoding)
    src = job.fmt.path
    out = Path(out_csv) if out_csv is not None else \
        src.with_name(f"{src.stem}_long.csv")
    _check_output(out, [src])
    if out.is_file() and not overwrite_existing:
        if verbose:
            print(f"[reshape] {out} exists; leaving it (overwrite_existing=False)")
        return out
    announce(on_progress, f"reading {src.name}")
    job.scan()
    report = job.report()
    announce(on_progress, f"writing {out.name}")
    ticker = Ticker(on_progress, total=job.reader.rows)
    written = 0
    with atomic_write(out, "w", newline="", encoding=OUTPUT_ENCODING) as fh:
        w = csv.writer(fh)
        w.writerow(report.columns_out)
        for n, row in enumerate(_Reader(job.fmt), 1):
            head = [str(n)] if job.row_col else [row[i] for i in job.id_pos]
            for name, pos in zip(job.names, job.value_pos):
                value = row[pos]
                if job.drop_blank and not value.strip():
                    continue
                w.writerow(head + [name, value])
                written += 1
            if n % _TICK_EVERY == 0:
                ticker.tick(_TICK_EVERY)
    if written != report.rows_out:
        raise RuntimeError(f"wide_to_long wrote {written} rows but planned "
                           f"{report.rows_out}")
    if verbose:
        for line in report.sentences():
            print(f"[reshape] {line}")
        print(f"[reshape] {src.name} -> {out} ({written:,} rows)")
    return out

taters.helpers.tables

A table is a table: one way to say where one lives, and one way to read or write it, whether that is a CSV file or a table in a database.

Everything in Taters that produces or consumes a table used to take a Path. A :class:TableRef is the one value that replaces it. For a file it is path-like -- open(), Path() and os.fspath() all work, so nothing that still thinks in files has to change -- and for a database table it is db://<connection>/<database>/<logical/path>, which is not path-like on purpose: a call site that has not learned about databases fails with a TypeError the moment it is handed one, instead of quietly creating a folder called db: and writing a CSV into it.

The canonical string form goes everywhere a path string went: YAML presets, {{save_as}} artifacts, lists of feature tables, command-line flags. A trailing .csv on a database address is dropped, so a template written as {{var:features_dir}}/readability.csv renders either way.

TableRef

TableRef(path=None, *, profile='', database='', logical='')

Where a table lives. A file path, or db://<connection>/<database>/<logical>.

Only the file kind is path-like. database is the database on that connection (main for a SQLite file, which only has the one); logical is the slash-separated name inside it (taters/study7/features/readability); the physical table name is derived from it by the database layer.

Source code in src\taters\helpers\tables.py
65
66
67
68
69
70
71
72
73
def __init__(self, path: Optional[Path] = None, *, profile: str = "",
             database: str = "", logical: str = ""):
    if path is None and not (profile and database and logical):
        raise ValueError("a TableRef is a path, or a connection, a database "
                         "and a logical name")
    self._path = Path(path) if path is not None else None
    self.profile = profile
    self.database = database
    self.logical = logical.strip("/")

name property

name

What to call the table in a sentence: a file's name with its extension, a database table's logical name.

root property

root

db://<connection>/<database> for a database table: the place every table of a run shares. A file's folder.

stem property

stem

The table's name: a file's stem, or the last logical segment.

local_dir

local_dir(work_dir=None)

A folder on disk for things that are not rows: a model file, a figure, a report fragment beside this table. For a file it is the file's folder; for a database table it is db_local/<connection>/ <database>/<logical parent> under the work folder.

Source code in src\taters\helpers\tables.py
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
def local_dir(self, work_dir: Optional[Path] = None) -> Path:
    """
    A folder on disk for things that are not rows: a model file, a
    figure, a report fragment beside this table. For a file it is the
    file's folder; for a database table it is ``db_local/<connection>/
    <database>/<logical parent>`` under the work folder.
    """
    if self._path is not None:
        return self._path.parent
    base = Path(work_dir) if work_dir is not None else Path.cwd()
    # a database name is whatever the server allowed; on disk it is a
    # folder name, so anything a folder cannot hold becomes _
    folder = re.sub(r"[^A-Za-z0-9_.-]", "_", self.database)
    home = base / "db_local" / self.profile / folder
    return home / Path(*self.parts[:-1]) if len(self.parts) > 1 else home

sibling

sibling(stem, suffix='.csv')

Another table beside this one: features/x.csv -> features/y.csv.

Source code in src\taters\helpers\tables.py
131
132
133
134
135
def sibling(self, stem: str, suffix: str = ".csv") -> "TableRef":
    """Another table beside this one: ``features/x.csv`` -> ``features/y.csv``."""
    if self._path is not None:
        return TableRef(self._path.with_name(stem + (suffix if suffix else "")))
    return self.under("/".join([*self.parts[:-1], stem]))

under

under(logical)

Another table in the same database, by its logical name.

Source code in src\taters\helpers\tables.py
127
128
129
def under(self, logical: str) -> "TableRef":
    """Another table in the same database, by its logical name."""
    return TableRef(profile=self.profile, database=self.database, logical=logical)

append_table

append_table(
    ref,
    fieldnames,
    *,
    key="text_id",
    encoding="utf-8-sig",
    settings=None,
    hints=None,
    finish=None
)

Open a table for writing that can be picked up where it stopped.

For a step whose rows each cost seconds of a model's time, "retried rather than resumed" (:func:atomic_write) throws hours away. This keeps the scratch file instead: rows go to <name>.part with a flush after every batch, a sidecar <stem>_partial.json records the settings that produced them, and on the next run the finished keys are read back so the caller can skip them. The file still appears under its real name only once complete. A sidecar that disagrees with the settings, or a header that differs, starts over.

Yields (writer, done): the writer has flush(), and done is the set of keys already in the scratch file. A database table has no scratch to keep, so it is written whole through :func:write_table and done is empty.

finish, when given, is the last pass: called with the header and the completed rows (a stream of dicts, in order) once every row is in, it returns the final header and rows, and those become the table -- columns worked out from the whole run, added to it without the rows ever being held in memory. A file streams its scratch into the final file; a database table is staged beside the final one and dropped after, so no text touches the local disk. A finish that fails on a file leaves the completed rows in the scratch for the next run (a database table restarts, as it always does).

Source code in src\taters\helpers\tables.py
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
@contextmanager
def append_table(ref: RefLike, fieldnames: Sequence[str], *, key: str = "text_id",
                 encoding: str = "utf-8-sig", settings: Optional[Mapping[str, Any]] = None,
                 hints: Optional[Mapping[str, str]] = None,
                 finish: Optional[Callable[[List[str], Iterator[Dict[str, str]]],
                                           Tuple[List[str], Iterator[Any]]]] = None):
    """
    Open a table for writing that can be picked up where it stopped.

    For a step whose rows each cost seconds of a model's time, "retried
    rather than resumed" (:func:`atomic_write`) throws hours away. This
    keeps the scratch file instead: rows go to ``<name>.part`` with a
    flush after every batch, a sidecar ``<stem>_partial.json`` records the
    ``settings`` that produced them, and on the next run the finished keys
    are read back so the caller can skip them. The file still appears
    under its real name only once complete. A sidecar that disagrees with
    the settings, or a header that differs, starts over.

    Yields ``(writer, done)``: the writer has ``flush()``, and ``done`` is
    the set of keys already in the scratch file. A database table has no
    scratch to keep, so it is written whole through :func:`write_table`
    and ``done`` is empty.

    ``finish``, when given, is the last pass: called with the header and
    the completed rows (a stream of dicts, in order) once every row is in,
    it returns the final header and rows, and those become the table --
    columns worked out from the whole run, added to it without the rows
    ever being held in memory. A file streams its scratch into the final
    file; a database table is staged beside the final one and dropped
    after, so no text touches the local disk. A ``finish`` that fails on
    a file leaves the completed rows in the scratch for the next run (a
    database table restarts, as it always does).
    """
    r = parse_ref(ref)
    if r.is_db:
        target = r.sibling(f"{r.stem}__stage") if finish is not None else r
        with write_table(target, fieldnames, encoding=encoding, hints=hints) as writer:
            writer.flush = lambda: None  # type: ignore[attr-defined]
            writer.kept = []  # type: ignore[attr-defined]
            yield writer, set()
        if finish is not None:
            try:
                _finish_into(r, finish, [str(c) for c in fieldnames],
                             read_rows(target), encoding=encoding, hints=hints)
            finally:
                delete_table(target)
        return
    from .atomic import SCRATCH_SUFFIX

    path = r.path
    path.parent.mkdir(parents=True, exist_ok=True)
    scratch = path.with_name(path.name + SCRATCH_SUFFIX)
    side = path.with_name(f"{path.stem}_partial.json")
    names = [str(c) for c in fieldnames]
    wanted = json.dumps(dict(settings or {}), sort_keys=True, default=str)
    kept: List[Dict[str, Any]] = []
    done: set = set()
    if scratch.is_file() and side.is_file():
        try:
            same = side.read_text(encoding="utf-8") == wanted
        except OSError:
            same = False
        if same:
            try:
                with scratch.open("r", newline="", encoding=encoding) as fh:
                    reader = csv.reader(fh)
                    header = next(reader, None)
                    if header == names:
                        for cells in reader:
                            # a torn last line has too few cells; drop it
                            if len(cells) != len(names):
                                continue
                            row = dict(zip(names, cells))
                            if row.get(key, "") in done:
                                continue
                            done.add(row.get(key, ""))
                            kept.append(row)
            except (OSError, csv.Error):
                kept, done = [], set()
    with scratch.open("w", newline="", encoding=encoding) as fh:
        out = csv.DictWriter(fh, fieldnames=names)
        out.writeheader()
        out.writerows(kept)
    side.write_text(wanted, encoding="utf-8")
    writer = _Writer(names)
    fh = scratch.open("a", newline="", encoding=encoding)
    out = csv.DictWriter(fh, fieldnames=names)
    writer._sink = lambda cells: out.writerow(dict(zip(names, cells)))
    writer.rows = len(kept)
    writer.kept = kept  # type: ignore[attr-defined]
    writer.flush = fh.flush  # type: ignore[attr-defined]
    try:
        yield writer, done
    except BaseException:
        # the whole point: the rows written so far stay for the next run
        fh.close()
        raise
    fh.close()
    if finish is None:
        os.replace(scratch, path)
    else:
        # the final file is written through the same scratch name, so the
        # completed rows step aside first -- and back if the pass fails, for
        # the next run to pick up
        aside = path.with_name(path.name + ".answered")
        os.replace(scratch, aside)
        try:
            with aside.open("r", newline="", encoding=encoding) as rf:
                rows = ({k: (v or "") for k, v in row.items() if k is not None}
                        for row in csv.DictReader(rf))
                _finish_into(r, finish, names, rows, encoding=encoding, hints=hints)
        except BaseException:
            os.replace(aside, scratch)
            raise
        aside.unlink(missing_ok=True)
    side.unlink(missing_ok=True)

count

count(ref, *, encoding='utf-8-sig', on_progress=None)

How many rows, by the cheapest honest means.

Source code in src\taters\helpers\tables.py
300
301
302
303
304
305
306
307
308
309
def count(ref: RefLike, *, encoding: str = "utf-8-sig",
          on_progress: Optional[Callable[..., None]] = None) -> int:
    """How many rows, by the cheapest honest means."""
    r = parse_ref(ref)
    if r.is_db:
        entry = _entry(r)
        return entry.n_rows if entry is not None else 0
    from .progress import count_rows

    return count_rows(r.path, encoding=encoding, on_progress=on_progress)

delete_table

delete_table(ref)

Remove a table Taters made, and whatever rides beside it: a file and its sidecars, or a database table with its registry and sidecar rows. A table Taters did not make is left alone. Returns whether anything was removed.

Source code in src\taters\helpers\tables.py
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
def delete_table(ref: RefLike) -> bool:
    """
    Remove a table Taters made, and whatever rides beside it: a file and
    its sidecars, or a database table with its registry and sidecar rows.
    A table Taters did not make is left alone. Returns whether anything
    was removed.
    """
    r = parse_ref(ref)
    if r.is_db:
        from . import db

        entry = _entry(r)
        if entry is None:
            return False
        db.drop_table(_session(r), entry)
        return True
    if not r.path.is_file():
        return False
    r.path.unlink()
    for kind in ("settings", "sets", "unscored"):
        _sidecar_file(r, kind).unlink(missing_ok=True)
    return True

export_table

export_table(
    *,
    source,
    out_csv,
    connection="",
    columns=None,
    overwrite_existing=False,
    verbose=True,
    on_progress=None
)

Copy a database table, or a join of tables, into a CSV file.

Parameters:

Name Type Description Default
source str or mapping

db://<connection>/<database>/<table> for one table, or a source spec (a base table plus joins; see :func:taters.helpers.sources.render) for several. A plain table name works when connection is given.

required
out_csv str or Path

Where the spreadsheet goes.

required
connection str

The connection a bare table name or a spec belongs to.

''
columns sequence of str

Only these columns, in this order. All of them by default.

None
overwrite_existing bool

Rebuild the file if it is already there.

False
verbose bool

Print what was copied.

True
on_progress callable

Progress callback.

None

Returns:

Type Description
Path

out_csv.

Source code in src\taters\helpers\tables.py
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
@records_settings(
    binding=("source", "connection", "columns"),
    outputs=("out_csv",),
    redo_without_record=True)
def export_table(
    *,
    source: Any,
    out_csv: Any,
    connection: str = "",
    columns: Optional[Sequence[str]] = None,
    overwrite_existing: bool = False,
    verbose: bool = True,
    on_progress: Optional[Callable[..., None]] = None,
) -> Path:
    """
    Copy a database table, or a join of tables, into a CSV file.

    Parameters
    ----------
    source : str or mapping
        ``db://<connection>/<database>/<table>`` for one table, or a source spec (a base
        table plus joins; see :func:`taters.helpers.sources.render`) for
        several. A plain table name works when ``connection`` is given.
    out_csv : str or Path
        Where the spreadsheet goes.
    connection : str, optional
        The connection a bare table name or a spec belongs to.
    columns : sequence of str, optional
        Only these columns, in this order. All of them by default.
    overwrite_existing : bool, default False
        Rebuild the file if it is already there.
    verbose : bool, default True
        Print what was copied.
    on_progress : callable, optional
        Progress callback.

    Returns
    -------
    pathlib.Path
        ``out_csv``.
    """
    from . import db, sources

    out = Path(out_csv)
    if out.is_file() and not overwrite_existing:
        if verbose:
            print(f"[export] {out} exists; leaving it (overwrite_existing=False)")
        return out
    spec = sources.coerce(source, connection=connection)
    profile = db.get_profile(spec.connection)
    if profile.export == "forbidden":
        raise db.NotPermitted(f"{profile.name} is marked as not to be copied out of "
                              f"(export: forbidden in its connection settings)")
    session = _session(sources.anchor(spec))
    with sources.guarded(spec, session):
        sql, params = sources.render(spec, session)
        announce(on_progress, f"reading from {spec.connection}")
        total = session.one(f"SELECT COUNT(*) FROM ({sql}) AS q", params)
        ticker = Ticker(on_progress, total=int(total or 0))
        seen = [0]

        def on_rows(n: int) -> None:
            ticker.tick(n - seen[0])
            seen[0] = n
        names, rows = db.stream_query(session, sql, params, on_rows=on_rows)
        wanted = [str(c) for c in columns] if columns else names
        missing = [c for c in wanted if c not in names]
        if missing:
            raise ValueError(f"the query has no column {missing[0]!r}; it has: "
                             f"{', '.join(names)}")
        with write_table(out, wanted) as w:
            for row in rows:
                w.writerow([row[c] for c in wanted])
    if verbose:
        print(f"[export] {spec.describe()} -> {out} ({w.rows:,} rows, {len(wanted)} columns)")
    return out

fingerprint

fingerprint(ref, *, encoding='utf-8-sig', delimiter=',')

The table's fingerprint from its rows (see db.Fingerprint).

Source code in src\taters\helpers\tables.py
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
def fingerprint(ref: RefLike, *, encoding: str = "utf-8-sig",
                delimiter: str = ",") -> str:
    """The table's fingerprint from its rows (see ``db.Fingerprint``)."""
    from . import db

    r = parse_ref(ref)
    if r.is_db:
        entry = _entry(r)
        if entry is None:
            raise FileNotFoundError(f"{r} is not in the database")
        return db.fingerprint_table(_session(r), entry)
    cols = columns_of(r, encoding=encoding, delimiter=delimiter)
    fp = db.Fingerprint(cols)
    for row in read_rows(r, encoding=encoding, delimiter=delimiter):
        fp.add(row.get(c, "") for c in cols)
    return fp.hexdigest()

handle

handle(ref)

What an analyzer returns: the Path for a file (unchanged for every caller that ever read one), the ref itself for a database table.

Source code in src\taters\helpers\tables.py
737
738
739
740
741
def handle(ref: RefLike):
    """What an analyzer returns: the ``Path`` for a file (unchanged for every
    caller that ever read one), the ref itself for a database table."""
    r = parse_ref(ref)
    return r.path if not r.is_db else r

import_table

import_table(
    *,
    csv_path,
    out,
    unique_key="",
    delimiter=None,
    encoding=None,
    overwrite_existing=False,
    verbose=True,
    on_progress=None
)

Copy a spreadsheet into a database table.

Parameters:

Name Type Description Default
csv_path str or Path

The spreadsheet. Its encoding and separator are detected unless given.

required
out str

db://<connection>/<database>/<table>.

required
unique_key str

A column that identifies each row; it becomes the table's key, and a repeated value stops the write.

''
delimiter str

How to read the file; detected when omitted.

None
encoding str

How to read the file; detected when omitted.

None
overwrite_existing bool

Replace a table Taters made earlier under this name.

False
verbose bool

Print what was copied.

True
on_progress callable

Progress callback.

None

Returns:

Type Description
TableRef

out.

Source code in src\taters\helpers\tables.py
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
@records_settings(
    binding=("csv_path", "delimiter", "encoding", "unique_key"),
    outputs=("out",),
    redo_without_record=True)
def import_table(
    *,
    csv_path: Any,
    out: Any,
    unique_key: str = "",
    delimiter: Optional[str] = None,
    encoding: Optional[str] = None,
    overwrite_existing: bool = False,
    verbose: bool = True,
    on_progress: Optional[Callable[..., None]] = None,
) -> TableRef:
    """
    Copy a spreadsheet into a database table.

    Parameters
    ----------
    csv_path : str or Path
        The spreadsheet. Its encoding and separator are detected unless given.
    out : str
        ``db://<connection>/<database>/<table>``.
    unique_key : str, optional
        A column that identifies each row; it becomes the table's key, and a
        repeated value stops the write.
    delimiter, encoding : str, optional
        How to read the file; detected when omitted.
    overwrite_existing : bool, default False
        Replace a table Taters made earlier under this name.
    verbose : bool, default True
        Print what was copied.
    on_progress : callable, optional
        Progress callback.

    Returns
    -------
    TableRef
        ``out``.
    """
    from . import reshape

    r = parse_ref(out)
    if not r.is_db:
        raise ValueError("out names a database table, db://<connection>/<database>/<table>")
    if exists(r) and not overwrite_existing:
        if verbose:
            print(f"[import] {r} exists; leaving it (overwrite_existing=False)")
        return r
    fmt = reshape.detect_format(csv_path, encoding=encoding, delimiter=delimiter)
    announce(on_progress, f"reading {fmt.name}")
    total = count(fmt.path, encoding=fmt.encoding)
    ticker = Ticker(on_progress, total=total)
    seen = [0]

    def on_rows(n: int) -> None:
        ticker.tick(n - seen[0])
        seen[0] = n
    with write_table(r, list(fmt.columns), unique_key=unique_key) as w:
        for row in reshape._Reader(fmt, on_rows=on_rows):
            w.writerow(row)
    if verbose:
        print(f"[import] {fmt.name} -> {r} ({w.rows:,} rows, {len(fmt.columns)} columns)")
    return r

is_ref

is_ref(value)

Whether value could name a table: a path-like or a db address.

Source code in src\taters\helpers\tables.py
180
181
182
183
def is_ref(value: Any) -> bool:
    """Whether ``value`` could name a table: a path-like or a db address."""
    return isinstance(value, (TableRef, Path)) or (
        isinstance(value, str) and bool(value.strip()))

modified_at

modified_at(ref)

When the table was last written, as a timestamp; None when absent.

Source code in src\taters\helpers\tables.py
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
def modified_at(ref: RefLike) -> Optional[float]:
    """When the table was last written, as a timestamp; None when absent."""
    import datetime as _dt

    r = parse_ref(ref)
    if not r.is_db:
        try:
            return r.path.stat().st_mtime
        except OSError:
            return None
    entry = _entry(r)
    if entry is None or not entry.written_at:
        return None
    try:
        return _dt.datetime.fromisoformat(entry.written_at).timestamp()
    except ValueError:
        return None

output_ref

output_ref(given, *, source, default)

Where an analyzer's table goes: what the caller said, or the default beside the source. default is the folder path under the run folder (("features", "readability")); the table takes the source's name. A database source defaults to the same database, under taters/<default...>/<name>, so anything in gives anything out only when somebody asks for it.

Source code in src\taters\helpers\tables.py
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
def output_ref(given: Any, *, source: RefLike, default: Sequence[str]) -> TableRef:
    """
    Where an analyzer's table goes: what the caller said, or the default
    beside the source. ``default`` is the folder path under the run folder
    (``("features", "readability")``); the table takes the source's name.
    A database source defaults to the same database, under
    ``taters/<default...>/<name>``, so anything in gives anything out only
    when somebody asks for it.
    """
    if given:
        return parse_ref(given)
    src = parse_ref(source)
    if src.is_db:
        return src.under("/".join(["taters", *default, src.stem]))
    return TableRef(Path.cwd().joinpath(*default) / f"{src.stem}.csv")

parse_ref

parse_ref(value)

A :class:TableRef from a path, a string, or a ref.

Accepts the db:/connection/... spelling too, which is what a db:// string becomes after one careless Path() on the way.

Source code in src\taters\helpers\tables.py
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
def parse_ref(value: RefLike) -> TableRef:
    """
    A :class:`TableRef` from a path, a string, or a ref.

    Accepts the ``db:/connection/...`` spelling too, which is what a
    ``db://`` string becomes after one careless ``Path()`` on the way.
    """
    if isinstance(value, TableRef):
        return value
    if isinstance(value, Path):
        text = str(value)
        if not (text.startswith("db:/") or text.startswith("db:\\")):
            return TableRef(value)
    elif isinstance(value, str):
        text = value.strip()
    else:
        raise TypeError(f"not a table: {value!r}")
    if text.startswith(DB_SCHEME) or text.startswith("db:/") or text.startswith("db:\\"):
        rest = text[3:].replace("\\", "/").lstrip("/")
        profile, _, tail = rest.partition("/")
        database, _, logical = tail.partition("/")
        if logical.lower().endswith(".csv"):
            logical = logical[:-4]
        if not profile or not database or not logical:
            raise ValueError(f"a database table is db://<connection>/<database>/<name>, "
                             f"not {text!r}")
        # the names become folders under the run folder for the things that
        # are not rows, so none of them can be allowed to climb out
        if not re.match(r"^[a-z][a-z0-9_-]*$", profile):
            raise ValueError(f"{profile!r} is not a connection name (lowercase letters, "
                             f"digits, _ or -)")
        if database in (".", "..") or re.search(r"[\x00-\x1f]", database):
            raise ValueError(f"{database!r} is not a database name")
        parts = logical.split("/")
        if any(not seg or seg in (".", "..") for seg in parts):
            raise ValueError(f"a table's name cannot have empty or '..' parts: {text!r}")
        return TableRef(profile=profile, database=database, logical=logical)
    return TableRef(Path(text))

read_cells

read_cells(
    ref,
    *,
    columns=None,
    encoding="utf-8-sig",
    delimiter=",",
    on_rows=None
)

Every row as a list of text cells, in columns order (the header's when none are named). The positional twin of :func:read_rows, for the matrix-shaped readers (a document-term matrix, a PCA) where a dict per row of five thousand cells is most of the work.

Source code in src\taters\helpers\tables.py
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
def read_cells(ref: RefLike, *, columns: Optional[Sequence[str]] = None,
               encoding: str = "utf-8-sig", delimiter: str = ",",
               on_rows: Optional[Callable[[int], None]] = None
               ) -> Iterator[List[str]]:
    """
    Every row as a list of text cells, in ``columns`` order (the header's
    when none are named). The positional twin of :func:`read_rows`, for the
    matrix-shaped readers (a document-term matrix, a PCA) where a dict per
    row of five thousand cells is most of the work.
    """
    r = parse_ref(ref)
    if r.is_db:
        from . import db

        entry = _entry(r)
        if entry is None:
            raise FileNotFoundError(f"{r} is not in the database")
        yield from db.read_cells(_session(r), entry, columns=columns, on_rows=on_rows)
        return
    n = 0
    with r.path.open("r", newline="", encoding=encoding) as fh:
        reader = csv.reader(fh, delimiter=delimiter)
        header = [str(c) for c in (next(reader, None) or [])]
        if columns is None:
            width = len(header)
            pick = None
        else:
            wanted = [str(c) for c in columns]
            missing = [c for c in wanted if c not in header]
            if missing:
                raise ValueError(f"{r} has no column {missing[0]!r}")
            pick = [header.index(c) for c in wanted]
        for row in reader:
            n += 1
            if pick is None:
                cells = [(v or "") for v in row]
                if len(cells) < width:
                    cells += [""] * (width - len(cells))
                yield cells[:width]
            else:
                yield [(row[i] if i < len(row) else "") or "" for i in pick]
            if on_rows is not None and n % 2000 == 0:
                on_rows(n)
    if on_rows is not None:
        on_rows(n)

read_rows

read_rows(
    ref,
    *,
    columns=None,
    encoding="utf-8-sig",
    delimiter=",",
    on_rows=None
)

Every row as a dict of text cells, in the table's own order.

Source code in src\taters\helpers\tables.py
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
def read_rows(ref: RefLike, *, columns: Optional[Sequence[str]] = None,
              encoding: str = "utf-8-sig", delimiter: str = ",",
              on_rows: Optional[Callable[[int], None]] = None
              ) -> Iterator[Dict[str, str]]:
    """Every row as a dict of text cells, in the table's own order."""
    r = parse_ref(ref)
    if r.is_db:
        from . import db

        entry = _entry(r)
        if entry is None:
            raise FileNotFoundError(f"{r} is not in the database")
        yield from db.read_rows(_session(r), entry, columns=columns, on_rows=on_rows)
        return
    n = 0
    with r.path.open("r", newline="", encoding=encoding) as fh:
        for row in csv.DictReader(fh, delimiter=delimiter):
            n += 1
            if columns is not None:
                row = {c: row.get(c) or "" for c in columns}
            else:
                row = {k: (v or "") for k, v in row.items() if k is not None}
            yield row
            if on_rows is not None and n % 2000 == 0:
                on_rows(n)
    if on_rows is not None:
        on_rows(n)

read_sidecar

read_sidecar(ref, kind)

<stem>_<kind>.json beside a file, or the database's sidecar row.

Source code in src\taters\helpers\tables.py
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
def read_sidecar(ref: RefLike, kind: str) -> Optional[dict]:
    """``<stem>_<kind>.json`` beside a file, or the database's sidecar row."""
    import json

    r = parse_ref(ref)
    if r.is_db:
        from . import db

        entry = _entry(r)
        return db.read_sidecar(_session(r), entry.table_name, kind) if entry else None
    try:
        with _sidecar_file(r, kind).open("r", encoding="utf-8") as fh:
            doc = json.load(fh)
    except (OSError, ValueError, UnicodeDecodeError):
        return None
    return doc if isinstance(doc, dict) else None

read_str_table

read_str_table(ref, *, encoding='utf-8-sig', columns=None)

The whole table as an all-string DataFrame (what the statistics read), or only columns of it, in the table's own order.

Source code in src\taters\helpers\tables.py
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
def read_str_table(ref: RefLike, *, encoding: str = "utf-8-sig",
                   columns: Optional[Sequence[str]] = None):
    """The whole table as an all-string DataFrame (what the statistics
    read), or only ``columns`` of it, in the table's own order."""
    import pandas as pd

    r = parse_ref(ref)
    wanted = [str(c) for c in columns] if columns is not None else None
    if not r.is_db:
        return pd.read_csv(r.path, dtype=str, keep_default_na=False, encoding=encoding,
                           usecols=(lambda c: c in set(wanted)) if wanted is not None else None)
    cols = [c for c in columns_of(r) if wanted is None or c in set(wanted)]
    # in blocks, so the peak is one block plus the frame rather than every
    # row as a Python list beside the frame built from it
    frames = []
    block: List[List[str]] = []
    for cells in read_cells(r, columns=cols):
        block.append(cells)
        if len(block) >= 50_000:
            frames.append(pd.DataFrame(block, columns=cols, dtype=str))
            block = []
    if block:
        frames.append(pd.DataFrame(block, columns=cols, dtype=str))
    if not frames:
        return pd.DataFrame({c: pd.Series(dtype=str) for c in cols})
    return frames[0] if len(frames) == 1 else pd.concat(frames, ignore_index=True)

require_local

require_local(ref, what)

The path of a file ref, or a plain refusal for a database one.

Source code in src\taters\helpers\tables.py
783
784
785
786
787
788
789
def require_local(ref: RefLike, what: str) -> Path:
    """The path of a file ref, or a plain refusal for a database one."""
    r = parse_ref(ref)
    if r.is_db:
        raise TypeError(f"{what} works on spreadsheets only for now; {r} is a "
                        f"table in a database. Copy it out with Wrangle data first.")
    return r.path

tables_under

tables_under(root)

Every table Taters wrote under a database root such as db://lab/study7/taters/study7/; nothing for a folder (its files are its own). A root that is just the database, db://lab/study7/, lists every table Taters made in it.

Source code in src\taters\helpers\tables.py
768
769
770
771
772
773
774
775
776
777
778
779
780
def tables_under(root: RefLike) -> List[TableRef]:
    """Every table Taters wrote under a database root such as
    ``db://lab/study7/taters/study7/``; nothing for a folder (its files are
    its own). A root that is just the database, ``db://lab/study7/``, lists
    every table Taters made in it."""
    text = str(root).rstrip("/")
    r = parse_ref(text + "/x")
    if not r.is_db:
        return []
    from . import db

    prefix = "/".join(r.parts[:-1])
    return [r.under(e.logical) for e in db.registered_under(_session(r), prefix)]

write_table

write_table(
    ref,
    fieldnames,
    *,
    encoding="utf-8-sig",
    hints=None,
    on_rows=None,
    fingerprint=False,
    unique_key=""
)

Open a table for writing; it appears only once complete.

For a file this is exactly atomic_write plus csv.DictWriter -- the bytes are what they always were. For a database table the rows are loaded, typed and swapped in by :func:taters.helpers.db.write_rows. hints names columns whose type is known ("text", "int", "float"). The writer's fingerprint is set on exit when asked for (a database table always has one; a file's costs a second pass over the cells, so it is opt-in).

Source code in src\taters\helpers\tables.py
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
@contextmanager
def write_table(ref: RefLike, fieldnames: Sequence[str], *,
                encoding: str = "utf-8-sig",
                hints: Optional[Mapping[str, str]] = None,
                on_rows: Optional[Callable[[int], None]] = None,
                fingerprint: bool = False,
                unique_key: str = ""):
    """
    Open a table for writing; it appears only once complete.

    For a file this is exactly ``atomic_write`` plus ``csv.DictWriter`` --
    the bytes are what they always were. For a database table the rows are
    loaded, typed and swapped in by :func:`taters.helpers.db.write_rows`.
    ``hints`` names columns whose type is known (``"text"``, ``"int"``,
    ``"float"``). The writer's ``fingerprint`` is set on exit when asked
    for (a database table always has one; a file's costs a second pass over
    the cells, so it is opt-in).
    """
    r = parse_ref(ref)
    writer = _Writer(fieldnames)
    if r.is_db:
        from . import db

        loader = db.Loader(_session(r, "write"), r.logical, writer.fieldnames, hints=hints,
                           on_rows=on_rows, unique_key=unique_key)
        writer._sink = loader.add
        try:
            yield writer
        except BaseException:
            loader.abort()
            raise
        entry = loader.finish()
        writer.fingerprint = entry.fingerprint
        return
    from .db import Fingerprint

    fp = Fingerprint(writer.fieldnames) if fingerprint else None
    with atomic_write(r.path, "w", newline="", encoding=encoding) as fh:
        out = csv.DictWriter(fh, fieldnames=writer.fieldnames)
        out.writeheader()

        def sink(cells: List[Any]) -> None:
            out.writerow(dict(zip(writer.fieldnames, cells)))
            if fp is not None:
                fp.add("" if v is None else str(v) for v in cells)
            if on_rows is not None and writer.rows % 2000 == 1999:
                on_rows(writer.rows + 1)
        writer._sink = sink
        yield writer
    if fp is not None:
        writer.fingerprint = fp.hexdigest()
    if on_rows is not None:
        on_rows(writer.rows)

taters.helpers.sources

A database source: one table, or a table with others joined on, as a thing a preset can hold and the wizard can read back before anything is run.

Not SQL text. A :class:Source is a base table plus zero or more joins on named keys, with the output columns named, and it is rendered into SQL for whichever engine the connection speaks -- from quoted identifiers only, every one checked against the catalog, so there is nothing to inject and a column that has been dropped since the pipeline was made is found before a step runs. Saved in YAML it reads like this::

source:
  from: {table: posts, as: p}
  join:
    - {table: users, as: u, how: left, on: [[p.user_id, u.id]]}
  columns: {id: p.id, body: p.body, age: u.age}
  order_by: [p.id]

A raw query: is the escape hatch for people who have SQL and know it is right; it is wrapped as a subquery and never edited.

:func:join_report is the read-back: for each join, the same counts the spreadsheet merge computes (matched, unmatched with examples, keys repeating on both sides, the rows the join would produce), from COUNT queries on the server, poured into the merge's own :class:MergeReport so the sentences a person sees are the ones they already know.

Source dataclass

Source(
    connection,
    database="",
    table="",
    alias="",
    joins=(),
    columns=dict(),
    order_by=(),
    query="",
    schema="",
)

tables property

tables

(alias, table) for the base and every join, in order.

describe

describe()

posts LEFT JOIN users LEFT JOIN outcomes (or a query).

Source code in src\taters\helpers\sources.py
105
106
107
108
109
110
def describe(self) -> str:
    """``posts LEFT JOIN users LEFT JOIN outcomes`` (or ``a query``)."""
    if self.query:
        return "a query"
    return " ".join([self.table] + [f"{j.how.upper()} JOIN {j.table}"
                                    for j in self.joins])

anchor

anchor(spec)

A :class:~taters.helpers.tables.TableRef for the source's base table (or query): what opens the right session, and what an output defaults beside. A spec that names no database gets the connection's saved default, or a refusal when there is none.

Source code in src\taters\helpers\sources.py
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
def anchor(spec: Source):
    """
    A :class:`~taters.helpers.tables.TableRef` for the source's base table
    (or ``query``): what opens the right session, and what an output
    defaults beside. A spec that names no database gets the connection's
    saved default, or a refusal when there is none.
    """
    from . import db
    from .tables import parse_ref

    database = spec.database or db.database_of(db.get_profile(spec.connection))
    if not database:
        raise ValueError(f"the source on {spec.connection} names no database "
                         f"(database: in the source spec)")
    return parse_ref(f"db://{spec.connection}/{database}/{spec.table or 'query'}")

coerce

coerce(value, *, connection='')

A :class:Source from a spec, a mapping, a db:// string or a table name.

Source code in src\taters\helpers\sources.py
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
def coerce(value: Any, *, connection: str = "") -> Source:
    """A :class:`Source` from a spec, a mapping, a ``db://`` string or a table name."""
    from .tables import DB_SCHEME, parse_ref

    if isinstance(value, Source):
        return value
    if isinstance(value, Mapping):
        return Source.from_dict(value, connection=connection)
    text = str(value).strip()
    if text.startswith(DB_SCHEME) or text.startswith("db:/"):
        ref = parse_ref(text)
        return Source(connection=ref.profile, database=ref.database, table=ref.logical)
    if not connection:
        raise ValueError(f"{text!r} needs a connection name")
    return Source(connection=connection, table=text)

expression_of

expression_of(spec, session, name)

The quoted "alias"."column" behind an output column name.

Source code in src\taters\helpers\sources.py
297
298
299
300
301
302
303
def expression_of(spec: Source, session, name: str) -> str:
    """The quoted ``"alias"."column"`` behind an output column name."""
    d = session.d
    for n, a, c in resolve_columns(spec, session):
        if n == name:
            return f"{d.quote(a)}.{d.quote(c)}" if a else d.quote(c)
    raise ValueError(f"the source has no column {name!r}")

guarded

guarded(spec, session)

The context a source's SQL runs in: read-only when the source is a typed query, nothing special when it is tables Taters rendered itself.

Source code in src\taters\helpers\sources.py
400
401
402
403
404
405
def guarded(spec: Source, session):
    """The context a source's SQL runs in: read-only when the source is a
    typed query, nothing special when it is tables Taters rendered itself."""
    import contextlib

    return session.read_only() if spec.query else contextlib.nullcontext()

join_report

join_report(spec, session, *, upto=None)

One :class:MergeReport per join: the left side is the base table with the earlier joins applied, the right side is the table being added.

Source code in src\taters\helpers\sources.py
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
def join_report(spec: Source, session, *, upto: Optional[int] = None
                ) -> List[MergeReport]:
    """
    One :class:`MergeReport` per join: the left side is the base table with
    the earlier joins applied, the right side is the table being added.
    """
    if spec.query:
        return []
    d = session.d
    reports: List[MergeReport] = []
    joins = spec.joins if upto is None else spec.joins[:upto]
    for k, j in enumerate(joins):
        left_from = _from_clause(spec, d, upto=k)
        right_from = f"FROM {d.qualified(j.table, j.schema or spec.schema)} AS {d.quote(j.alias)}"
        left_refs = [_split(a, spec.alias) for a, _b in j.on]
        right_refs = [_split(b, j.alias) for _a, b in j.on]
        # the left key may sit on the base or on an earlier join; each pair
        # names its own alias, and the side summary needs one alias to count
        # under, so the counts are expressed through the full left FROM
        left_keys = [f"{a}.{c}" for a, c in left_refs]
        right_keys = [c for _a, c in right_refs]
        lk = [f"{d.quote(a)}.{d.quote(c)}" for a, c in left_refs]
        rk = [f"{d.quote(j.alias)}.{d.quote(c)}" for c in right_keys]
        match = " AND ".join(f"{a} = {b}" for a, b in zip(lk, rk))
        left = _side_any(session, label=_left_label(spec, k), sql_from=left_from,
                         mine=lk, other_from=right_from, theirs=rk, keys=left_keys)
        right = _side_any(session, label=j.table, sql_from=right_from, mine=rk,
                          other_from=left_from, theirs=lk, keys=right_keys)
        inner = int(session.one(f"SELECT COUNT(*) {left_from} INNER JOIN "
                                f"{d.qualified(j.table, j.schema or spec.schema)} AS "
                                f"{d.quote(j.alias)} ON {match}") or 0)
        rows_out = {"inner": inner, "left": inner + (left.rows - left.rows_matched)}
        rows_out["outer"] = rows_out["left"] + (right.rows - right.rows_matched)
        m2m = _m2m(session, d, lk, rk, left_from, right_from, left_refs, right_keys)
        left_multi = _multi(session, d, lk, left_from, rk, right_from)
        right_multi = _multi(session, d, rk, right_from, lk, left_from)
        fan_out: List[Tuple[str, int]] = []
        if right_multi:
            rkeys = ", ".join(rk)
            for row in session.all(d.limit(
                    f"SELECT {rkeys}, COUNT(*) AS n {right_from} WHERE NOT "
                    f"{_blank_any(d, rk)} AND EXISTS (SELECT 1 {left_from} WHERE {match}) "
                    f"GROUP BY {rkeys} HAVING COUNT(*) > 1 ORDER BY n DESC", 3)):
                *key, n = row
                fan_out.append((" | ".join(str(v) for v in key), int(n)))
        if m2m["count"]:
            relationship = "many-to-many"
        elif right_multi:
            relationship = "one-to-many"
        elif left_multi:
            relationship = "many-to-one"
        else:
            relationship = "one-to-one"
        clashes, columns_out = _clashes(session, spec, k, j, match, left_from)
        reports.append(MergeReport(
            left=left, right=right, on=tuple(c for _a, c in left_refs),
            right_on=tuple(right_keys),
            key_match="exact", relationship=relationship,
            many_to_many_keys=m2m["count"], many_to_many_examples=tuple(m2m["examples"]),
            rows_out=rows_out, loose_extra_matches=0, loose_examples=(),
            clashes=clashes, columns_out=columns_out, what="table",
            fan_out_examples=tuple(fan_out)))
    return reports

query_problem

query_problem(query)

Why a typed query cannot be a source, or "" when it reads as one SELECT. The session it runs on is made read-only as well (see :func:guarded); this is the cheap first check that says so in words.

Source code in src\taters\helpers\sources.py
383
384
385
386
387
388
389
390
391
392
def query_problem(query: str) -> str:
    """Why a typed query cannot be a source, or "" when it reads as one
    SELECT. The session it runs on is made read-only as well (see
    :func:`guarded`); this is the cheap first check that says so in words."""
    text = _strip_comments(str(query)).strip()
    if ";" in text:
        return "a query source is one SELECT statement; ';' is not allowed in it"
    if not re.match(r"(?is)^\s*(select|with)\b", text):
        return "a query source has to start with SELECT (or WITH); it is read-only"
    return ""

render

render(spec, session, *, columns=None, order_by_exprs=None)

The SELECT for a source, from quoted identifiers only.

columns narrows the output to those names (in that order). order_by_exprs replaces the spec's own ordering with ready SQL expressions (the gather uses it to sort by its group keys). The result carries no parameters today, but the pair is the shape every caller handles, so a WHERE pushed down later changes nothing upstream.

Source code in src\taters\helpers\sources.py
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
def render(spec: Source, session, *, columns: Optional[Sequence[str]] = None,
           order_by_exprs: Optional[Sequence[str]] = None
           ) -> Tuple[str, Tuple[Any, ...]]:
    """
    The SELECT for a source, from quoted identifiers only.

    ``columns`` narrows the output to those names (in that order).
    ``order_by_exprs`` replaces the spec's own ordering with ready SQL
    expressions (the gather uses it to sort by its group keys). The result
    carries no parameters today, but the pair is the shape every caller
    handles, so a WHERE pushed down later changes nothing upstream.
    """
    d = session.d
    if spec.query:
        sql = f"SELECT * FROM ({spec.query}) AS q"
        if columns:
            sql = f"SELECT {', '.join(d.quote(c) for c in columns)} FROM ({spec.query}) AS q"
        return sql, ()
    resolved = resolve_columns(spec, session)
    if columns:
        by_name = {n: (a, c) for n, a, c in resolved}
        missing = [c for c in columns if c not in by_name]
        if missing:
            raise ValueError(f"the source has no column {missing[0]!r}")
        resolved = [(n, *by_name[n]) for n in columns]
    selects = ", ".join(f"{d.quote(a)}.{d.quote(c)} AS {d.quote(n)}" for n, a, c in resolved)
    sql = f"SELECT {selects} {_from_clause(spec, d)}"
    if order_by_exprs:
        sql += " ORDER BY " + ", ".join(order_by_exprs)
    elif spec.order_by:
        keys = []
        for ref in spec.order_by:
            a, c = _split(ref, spec.alias)
            keys.append(f"{d.quote(a)}.{d.quote(c)}")
        sql += " ORDER BY " + ", ".join(keys)
    return sql, ()

resolve_columns

resolve_columns(spec, session)

(output name, alias, column) for every output column, checked against the catalog. With no columns mapping: every column of every table, base first; a name present in several tables is kept bare for the first and prefixed <table>__ after that, merge's rule.

Source code in src\taters\helpers\sources.py
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
def resolve_columns(spec: Source, session) -> List[Tuple[str, str, str]]:
    """
    ``(output name, alias, column)`` for every output column, checked
    against the catalog. With no ``columns`` mapping: every column of every
    table, base first; a name present in several tables is kept bare for
    the first and prefixed ``<table>__`` after that, merge's rule.
    """
    if spec.query:
        with session.read_only():
            return [(c, "", c) for c in session.columns_of_query(spec.query)]
    known: Dict[str, List[str]] = {}
    for alias, table in spec.tables:
        try:
            info = session.d.describe(session, table, schema=spec.schema)
        except Exception:
            raise ValueError(f"{spec.connection} has no table called {table}") from None
        known[alias] = info.column_names
    out: List[Tuple[str, str, str]] = []
    if spec.columns:
        for name, ref in spec.columns.items():
            alias, col = _split(ref, spec.alias)
            if alias not in known:
                raise ValueError(f"{ref!r}: no table is called {alias!r} in this source")
            if col not in known[alias]:
                raise ValueError(f"{dict(spec.tables)[alias]} no longer has a column "
                                 f"called {col}")
            out.append((name, alias, col))
        return out
    # a join's right-hand key repeats the left one row for row, so it is
    # matched, not carried -- the same rule the spreadsheet merge follows
    matched = {(j.alias, _split(b, j.alias)[1]) for j in spec.joins for _a, b in j.on}
    taken: Dict[str, str] = {}
    for alias, table in spec.tables:
        for col in known[alias]:
            if (alias, col) in matched:
                continue
            name = col
            if name in taken:
                name = f"{table}{SEPARATOR}{col}"
                k = 2
                while name in taken:
                    name = f"{table}{SEPARATOR}{col}_{k}"
                    k += 1
            taken[name] = alias
            out.append((name, alias, col))
    return out

taters.helpers.db

Database connections: where they are, how to log in, what they allow, and how a table gets written into one.

Three engines behind one small seam. SQLite is the standard library and a real target for a single person's study; MySQL (and MariaDB) and PostgreSQL are what a lab's data lives in. Each is a :class:Dialect of a few dozen lines -- how to quote a name, what a placeholder looks like, how to list tables, how to swap one table for another atomically -- over the DB-API connection its driver gives. The drivers are imported inside functions, so importing Taters never imports a database driver, and a missing one is a sentence naming the extra rather than an import error at startup.

Profiles are the saved part: a name, the engine, where the server is, which database and which user. Never the password. That comes from TATERS_DB_PASSWORD_<NAME>, or from a masked prompt once per session, and lives in this process's memory and nowhere else -- not in the environment (a child process would inherit it), not in the run log, not in the manifest, not in settings.json (which gets synced). Pool workers never connect at all: the parent reads the rows and ships text to them.

Writing is copy-then-swap. Rows load into a scratch table, get their types, and the finished table replaces the old one in one rename, so a run killed halfway leaves the previous table exactly as it was. Every table Taters writes carries a hidden _taters_row so it reads back in the order it was written, and a row in _taters_tables saying what it is and a fingerprint of its rows, which is how provenance tells "unchanged" from "somebody edited this".

DbError

Bases: RuntimeError

Something the database said, in words a person can act on.

Dialect

What differs between engines. The base class is SQLite.

blank

blank(quoted)

True when the cell is NULL or empty after trimming.

Source code in src\taters\helpers\db.py
623
624
625
def blank(self, quoted: str) -> str:
    """True when the cell is NULL or empty after trimming."""
    return f"({quoted} IS NULL OR TRIM({self.as_text(quoted)}) = '')"

cast

cast(column, kind)

kind is "int" or "float"; non-finite words become NULL.

Source code in src\taters\helpers\db.py
580
581
582
583
584
585
586
def cast(self, column: str, kind: str) -> str:
    """``kind`` is ``"int"`` or ``"float"``; non-finite words become NULL."""
    q = self.quote(column)
    guard = (f"CASE WHEN LOWER(TRIM({q})) IN ('nan','inf','-inf','+inf',"
             f"'infinity','-infinity','+infinity') OR TRIM({q}) = '' THEN NULL ELSE ")
    target = self.int_type if kind == "int" else self.float_type
    return f"{guard}CAST({q} AS {target}) END"

count_tables

count_tables(session, schema='')

How many tables the account can see, in one query -- the probe wants the number, not a description of each.

Source code in src\taters\helpers\db.py
652
653
654
655
656
657
def count_tables(self, session: "Session", schema: str = "") -> int:
    """How many tables the account can see, in one query -- the probe
    wants the number, not a description of each."""
    return int(session.one("SELECT COUNT(*) FROM sqlite_master WHERE type IN "
                           "('table', 'view') AND name NOT LIKE 'sqlite_%' "
                           "AND name NOT LIKE '_taters_%'") or 0)

driver_errors

driver_errors()

The exception classes the driver raises for something the server said, so a session can turn them into words. Not a programming error: a TypeError from a bad parameter stays what it is.

Source code in src\taters\helpers\db.py
527
528
529
530
531
532
533
def driver_errors(self) -> tuple:
    """The exception classes the driver raises for something the server
    said, so a session can turn them into words. Not a programming
    error: a TypeError from a bad parameter stays what it is."""
    import sqlite3

    return (sqlite3.Error,)

list_databases

list_databases(session)

The databases on the server this account can see; none for a file.

Source code in src\taters\helpers\db.py
648
649
650
def list_databases(self, session: "Session") -> List[str]:
    """The databases on the server this account can see; none for a file."""
    return []

numeric

numeric(quoted)

True when a non-blank cell reads as a number. An offer, not a contract: the wizard shows the verdict and the arrows can change it.

Source code in src\taters\helpers\db.py
627
628
629
630
631
632
def numeric(self, quoted: str) -> str:
    """True when a non-blank cell reads as a number. An offer, not a
    contract: the wizard shows the verdict and the arrows can change it."""
    t = f"TRIM({self.as_text(quoted)})"
    return (f"(typeof({quoted}) IN ('integer', 'real') OR "
            f"({t} GLOB '*[0-9]*' AND NOT {t} GLOB '*[^0-9.eE+-]*'))")

sort_key

sort_key(quoted)

An expression that puts equal values together, blanks with NULLs.

Source code in src\taters\helpers\db.py
616
617
618
def sort_key(self, quoted: str) -> str:
    """An expression that puts equal values together, blanks with NULLs."""
    return f"COALESCE(CAST({quoted} AS TEXT), '')"

stream_scope

stream_scope(con)

What a streaming read has to run inside: nothing, for most engines.

Source code in src\taters\helpers\db.py
640
641
642
def stream_scope(self, con):
    """What a streaming read has to run inside: nothing, for most engines."""
    return contextlib.nullcontext()

swap

swap(session, name, new, old, schema='')

new becomes name; what was name becomes old. Atomic.

Source code in src\taters\helpers\db.py
588
589
590
591
592
593
594
595
596
597
598
599
600
def swap(self, session: "Session", name: str, new: str, old: str,
         schema: str = "") -> None:
    """``new`` becomes ``name``; what was ``name`` becomes ``old``. Atomic."""
    q = lambda n: self.qualified(n, schema)  # noqa: E731
    session.run("BEGIN IMMEDIATE")
    try:
        if self.table_exists(session, name, schema):
            session.run(f"ALTER TABLE {q(name)} RENAME TO {self.quote(old)}")
        session.run(f"ALTER TABLE {q(new)} RENAME TO {self.quote(name)}")
        session.run("COMMIT")
    except Exception:
        session.run("ROLLBACK")
        raise

DriverMissing

Bases: DbError

The engine's driver is not installed; the message names the extra.

Fingerprint

Fingerprint(columns)

A table's identity from its rows, independent of their order and of how the engine typed them.

Each row hashes to a digest; the table's fingerprint is a hash over the column names, the row count and the sum of the row digests. A sum rather than an exclusive-or, so two identical rows do not cancel out.

Source code in src\taters\helpers\db.py
413
414
415
416
def __init__(self, columns: Sequence[str]):
    self.columns = [str(c) for c in columns]
    self.rows = 0
    self._sum = 0

HostKey dataclass

HostKey(type, blob, fingerprint)

One key an ssh host presented: its type, the key itself (base64, as a known_hosts line carries it) and the fingerprint people compare.

HostKeyUnknown

HostKeyUnknown(message, profile, keys=())

Bases: DbError

The ssh host is not trusted yet. keys are what it presented (see :func:host_keys), so the person can be shown them and asked.

Source code in src\taters\helpers\db.py
119
120
121
122
def __init__(self, message: str, profile: "Profile", keys: Sequence["HostKey"] = ()):
    super().__init__(message)
    self.profile = profile
    self.keys = list(keys)

Loader

Loader(
    session,
    logical,
    columns,
    *,
    hints=None,
    on_rows=None,
    unique_key=""
)

Rows pushed in one at a time; a typed table swapped in at the end.

add takes a row, finish makes the table and returns its registry entry, abort drops the scratch. Nothing is buffered beyond one insert batch, so a table of any size costs the same memory.

hints names columns whose type is known in advance ("text", "int", "float"); when every column is hinted the rows go straight into the typed table. Otherwise they load as text and are copied with casts on the server, so a 500k-row table is never read back into Python to learn its types. Refuses to replace a table Taters did not make.

Source code in src\taters\helpers\db.py
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194
2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
def __init__(self, session: Session, logical: str, columns: Sequence[str], *,
             hints: Optional[Mapping[str, str]] = None,
             on_rows: Optional[Callable[[int], None]] = None,
             unique_key: str = ""):
    d, schema = session.d, session.profile.schema
    self.session, self.d, self.schema = session, d, schema
    self.logical = logical
    self.unique_key = str(unique_key or "")
    if self.unique_key and self.unique_key not in [str(c) for c in columns]:
        raise DbError(f"{logical} has no column {self.unique_key!r} to make its key")
    self.physical = sanitize_table_name(logical)
    self.columns = [str(c) for c in columns]
    self.on_rows = on_rows
    self.hints = {str(k): str(v) for k, v in (hints or {}).items()}
    # past the engine's column limit the measures go into one JSON
    # column and only the identity columns stay real. which ones are
    # "identity" has to be decided up front, before a row is seen: the
    # ids by name, and whatever the caller hinted is text
    self.packed = len(self.columns) + 1 > d.max_columns
    if self.packed:
        self.real = [c for c in self.columns
                     if c in ID_COLUMNS or self.hints.get(c) == "text"]
        if len(self.real) + 2 > d.max_columns:
            raise DbError(f"{logical} has {len(self.real):,} text columns, more "
                          f"than {session.profile.engine_label} can hold in one "
                          f"table, and text cannot be packed")
        if self.unique_key and self.unique_key not in self.real:
            raise DbError(f"{logical} is too wide for {session.profile.engine_label} "
                          f"to hold as columns, and its key {self.unique_key!r} "
                          f"would be packed away with the measures")
    else:
        self.real = list(self.columns)
    self.real_index = [self.columns.index(c) for c in self.real]
    self.packed_index = [i for i, c in enumerate(self.columns) if c not in set(self.real)]
    ensure_registry(session)
    old = registered(session, self.physical)
    if d.table_exists(session, self.physical, schema) and \
            (old is None or old.logical != logical):
        raise DbError(f"{session.profile.name} already has a table called "
                      f"{self.physical}"
                      + (" that Taters did not make" if old is None
                         else f" holding {old.logical}")
                      + "; it will not be replaced. Pick another name.")
    taken: set = {ROW_COLUMN, PACKED_COLUMN}
    self.physical_cols = [_column_name(c, i, taken) for i, c in enumerate(self.real)]
    self.all_hinted = all(c in self.hints for c in self.real)
    # scratch names nobody could have made on purpose: fixed suffixes
    # (`posts__tnew`) could be somebody's real table, and dropping those
    # "to clear the way" was dropping their data
    tmp = f"_taters_tmp_{_secrets.token_hex(4)}"
    self.load = tmp + "_load"
    self.new = tmp + "_new"
    self.told = tmp + "_old"
    self.swapped = False
    self.kinds = [self.hints.get(c, "text") for c in self.real]
    self.fp = Fingerprint(self.columns)
    self.tracker = _TypeTracker(len(self.real))
    target = self.new if self.all_hinted else self.load
    extra = [(PACKED_COLUMN, d.text_type)] if self.packed else []
    if self.all_hinted:
        d.create(session, self.new,
                 [(p, self._typed(k)) for p, k in zip(self.physical_cols, self.kinds)]
                 + extra, schema=schema)
    else:
        d.create(session, self.load,
                 [(p, d.text_type) for p in self.physical_cols] + extra, schema=schema)
    stored = [ROW_COLUMN, *self.physical_cols] + ([PACKED_COLUMN] if self.packed else [])
    self.names = ", ".join(d.quote(c) for c in stored)
    marks = ", ".join("?" for _ in stored)
    self.insert = (f"INSERT INTO {d.qualified(target, schema)} ({self.names}) "
                   f"VALUES ({marks})")
    self.batch: List[tuple] = []
    self.n = 0
    self.done = False

LoginHints dataclass

LoginHints(
    user="",
    password=None,
    socket="",
    source="",
    os_user="",
    problem="",
)

What the ssh host's own database client would do: the login saved in ~/.my.cnf / ~/.pgpass (user, password), the database's local socket if there is one, and the ssh user. Nothing here is shown or saved; a password found is used for the session and forgotten.

NotPermitted

Bases: DbError

The account cannot do what the step needs.

Probe dataclass

Probe(
    select=False,
    create=False,
    drop=False,
    registry=False,
    version="",
    tables=0,
    encrypted=None,
    notes=(),
)

sentences

sentences(profile)

The test's read-back, one line each.

Source code in src\taters\helpers\db.py
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
def sentences(self, profile: Profile) -> List[str]:
    """The test's read-back, one line each."""
    if not self.select:
        return [f"Could not read from {profile.name}: {'; '.join(self.notes) or 'no permission to select'}."]
    enc = {True: "yes", False: "no", None: "n/a (a file)"}[self.encrypted]
    head = (f"Connected to {profile.name} ({self.version}, {profile.where})"
            + (f" as {profile.user}" if profile.user else "") + f". Encrypted: {enc}.")
    if profile.dialect != "sqlite" and profile.database:
        head = (f"Connected to {profile.name} ({self.version}, {profile.server}) as "
                f"{profile.user}, on database {profile.database}. Encrypted: {enc}.")
    out = [head, f"Can read: yes — {self.tables} table(s).",
           f"Can create tables: {'yes' if self.create else 'no'}.",
           f"Can replace tables it made: {'yes' if self.drop else 'no'}."]
    if not self.can_write:
        out.append(f"Wrangle data can copy tables out of {profile.name} into "
                   f"spreadsheets, and a run can read from it; its results will "
                   f"have to go to files or to a database that allows new tables."
                   + (f" Ask whoever manages the database to grant CREATE on "
                      f"{profile.database}, or add a second connection that has it."
                      if profile.dialect != "sqlite" else ""))
    elif not self.drop:
        out.append("Runs work. A re-run writes new tables beside the old ones "
                   "instead of replacing them.")
    return out

Profile dataclass

Profile(
    name,
    dialect,
    host="",
    port=0,
    user="",
    database="",
    schema="",
    path="",
    ssl="required",
    ssl_ca="",
    password="ask",
    export="allowed",
    ssh_host="",
    ssh_port=0,
    ssh_user="",
    ssh_key="",
    ssh_hostkey="",
    socket="",
)

A saved way to reach a database. Everything but the password.

password records how the password is obtained -- "ask" (a masked prompt, once per session), "env" (TATERS_DB_PASSWORD_<NAME>), "server" (read from the person's own ~/.my.cnf or ~/.pgpass on the ssh host, each session) or "none" -- never the password itself. A password value that is anything else is dropped when the profile is loaded.

server property

server

db.lab.edu (with the port when it is not the usual one), or the socket or address as seen through the ssh tunnel.

ssh_address property

ssh_address

user@host:port for the tunnel, as a person would type it.

where property

where

study7 on db.lab.edu when pointed at a database, else just the server; or the file's name.

label

label()

One line for a picker: engine, who at where, and how it logs in. A connection is the server and the login; the database comes later.

Source code in src\taters\helpers\db.py
224
225
226
227
228
229
230
231
232
233
234
235
236
237
def label(self) -> str:
    """One line for a picker: engine, who at where, and how it logs in. A
    connection is the server and the login; the database comes later."""
    parts = [self.engine_label]
    if self.dialect == "sqlite":
        parts.append(self.where)
    else:
        parts.append(f"{self.user}@{self.server}" if self.user else self.server)
        parts.append({"ask": "asks for its password",
                      "env": f"password from {password_env(self.name)}",
                      "server": "password from your saved login on the ssh host",
                      "none": ("no password, by socket login" if self.socket
                               else "no password")}[self.password])
    return " · ".join(parts)

Registered dataclass

Registered(
    table_name,
    logical,
    layout,
    columns,
    n_rows,
    fingerprint,
    written_at,
    taters_version="",
)

One row of the registry, as Python.

table_columns property

table_columns

The columns the table really has, besides the row number: every physical column, plus the packed column when there is one.

Session

Session(profile, con, dialect)

One open connection, its dialect, and the small verbs the code uses.

Source code in src\taters\helpers\db.py
1704
1705
1706
1707
def __init__(self, profile: Profile, con, dialect: Dialect):
    self.profile = profile
    self.con = con
    self.d = dialect

read_only

read_only()

Run a block with the session unable to write: a query somebody typed into a YAML file gets exactly the rights a SELECT needs.

Source code in src\taters\helpers\db.py
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
@contextlib.contextmanager
def read_only(self):
    """Run a block with the session unable to write: a query somebody
    typed into a YAML file gets exactly the rights a SELECT needs."""
    self.run(self.d.read_only_on)
    try:
        yield self
    finally:
        try:
            self.run(self.d.read_only_off)
        except Exception:
            pass

stream

stream(sql, params=(), batch=_BATCH)

Rows one at a time without holding the result set.

Source code in src\taters\helpers\db.py
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
def stream(self, sql: str, params: Sequence[Any] = (),
           batch: int = _BATCH) -> Iterator[tuple]:
    """Rows one at a time without holding the result set."""
    with self.d.stream_scope(self.con):
        cur = self.d.stream_cursor(self.con)
        try:
            with self._said():
                cur.execute(self.d.translate(sql), tuple(params))
                while True:
                    chunk = cur.fetchmany(batch)
                    if not chunk:
                        break
                    for row in chunk:
                        yield tuple(row)
        finally:
            cur.close()

accept_host_keys

accept_host_keys(profile, keys, *, save=True)

Trust these keys for this host from now on: OpenSSH's way is a line per key in ~/.ssh/known_hosts, the same line ssh would have written after a "yes"; plink's is a fingerprint on the profile, handed to it as -hostkey every time. Returns the profile to use (changed only for plink).

Source code in src\taters\helpers\db.py
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
def accept_host_keys(profile: Profile, keys: Sequence[HostKey], *,
                     save: bool = True) -> Profile:
    """
    Trust these keys for this host from now on: OpenSSH's way is a line
    per key in ``~/.ssh/known_hosts``, the same line ``ssh`` would have
    written after a "yes"; plink's is a fingerprint on the profile, handed
    to it as ``-hostkey`` every time. Returns the profile to use (changed
    only for plink).
    """
    keys = [k for k in keys if k.fingerprint]
    if not keys:
        raise DbError(f"nothing to accept: {profile.ssh_host} showed no key")
    if uses_putty(profile):
        have = [f for f in profile.ssh_hostkey.split(",") if f.strip()]
        fps = have + [k.fingerprint for k in keys if k.fingerprint not in have]
        updated = replace(profile, ssh_hostkey=",".join(fps))
        if save:
            save_profile(updated)
        return updated
    port = profile.ssh_port or 22
    name = profile.ssh_host if port == 22 else f"[{profile.ssh_host}]:{port}"
    path = known_hosts_path()
    path.parent.mkdir(parents=True, exist_ok=True)
    try:
        path.parent.chmod(0o700)
    except OSError:
        pass
    existing = path.read_text(encoding="utf-8") if path.is_file() else ""
    lines = [f"{name} {k.type} {k.blob}" for k in keys if k.blob]
    if not lines:
        raise DbError(f"{profile.ssh_host}'s key could not be fetched to record it "
                      f"(ssh-keyscan is missing); log in once by hand with "
                      f"`{ssh_hint(profile)}` instead.")
    new = [ln for ln in lines if ln not in existing.splitlines()]
    if new:
        with path.open("a", encoding="utf-8") as fh:
            if existing and not existing.endswith("\n"):
                fh.write("\n")
            fh.write("\n".join(new) + "\n")
    return profile

cell_text

cell_text(value)

A database value as the text a CSV export of the same table would hold.

str() rather than isoformat() for dates, so 2024-01-01 12:00:00 comes out the way a spreadsheet export writes it and an id built from a query equals one built from a file. Floats use repr, the shortest text that reads back to the same number.

Source code in src\taters\helpers\db.py
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
def cell_text(value: Any) -> str:
    """
    A database value as the text a CSV export of the same table would hold.

    ``str()`` rather than ``isoformat()`` for dates, so ``2024-01-01
    12:00:00`` comes out the way a spreadsheet export writes it and an id
    built from a query equals one built from a file. Floats use ``repr``,
    the shortest text that reads back to the same number.
    """
    if value is None:
        return ""
    if isinstance(value, bool):
        return "1" if value else "0"
    if isinstance(value, float):
        return repr(value)
    if isinstance(value, (bytes, bytearray, memoryview)):
        return bytes(value).decode("utf-8", errors="replace")
    if isinstance(value, decimal.Decimal):
        return format(value, "f")
    if isinstance(value, (_dt.datetime, _dt.date, _dt.time)):
        return str(value)
    return str(value)

cell_token

cell_token(text)

One cell canonicalized by value, so a CSV and a database copy of the same rows fingerprint the same whatever the column's type: blank or non-finite is empty, an integer is i:<n>, another number is f:<repr(float)>, and anything else is the text itself.

Source code in src\taters\helpers\db.py
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
def cell_token(text: str) -> bytes:
    """
    One cell canonicalized by *value*, so a CSV and a database copy of the
    same rows fingerprint the same whatever the column's type: blank or
    non-finite is empty, an integer is ``i:<n>``, another number is
    ``f:<repr(float)>``, and anything else is the text itself.
    """
    s = text.strip()
    if not s or s.lower() in _NON_FINITE:
        return b"\x00"
    if _INTEGER.match(s):
        return b"i:" + str(int(s)).encode()
    if _NUMBER.match(s):
        try:
            f = float(s)
        except ValueError:
            return b"s:" + s.encode("utf-8")
        if not math.isfinite(f):
            return b"\x00"
        if f == int(f) and abs(f) < 2 ** 53:
            return b"i:" + str(int(f)).encode()
        return b"f:" + repr(f).encode()
    return b"s:" + s.encode("utf-8")

connect

connect(profile, password=None, ask=None, on_step=None)

Open a session. Refuses to run inside a pool worker: the parent reads the rows and ships text, so a worker never needs, or holds, a password. on_step hears what is being done as it is done -- a tunnel being opened, a login -- so a screen never just sits there.

Source code in src\taters\helpers\db.py
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
def connect(profile: Profile, password: Optional[str] = None,
            ask: Optional[Callable[[str], str]] = None,
            on_step: Optional[Callable[[str], None]] = None) -> Session:
    """
    Open a session. Refuses to run inside a pool worker: the parent reads
    the rows and ships text, so a worker never needs, or holds, a password.
    ``on_step`` hears what is being done as it is done -- a tunnel being
    opened, a login -- so a screen never just sits there.
    """
    if multiprocessing.parent_process() is not None:
        raise DbError("a database is read by the main process only; workers "
                      "receive rows from it")
    d = dialect_for(profile.dialect)
    if password is None:
        password = password_for(profile, ask)
    say = on_step or (lambda text: None)
    if profile.tunneled:
        tunnel = _TUNNELS.get(profile.name)
        if tunnel is None or tunnel[0].poll() is not None:
            say(f"Opening the ssh tunnel to {profile.ssh_address}")
            open_tunnel(profile)
    if profile.dialect == "sqlite":
        say(f"Opening {profile.where}")
    else:
        say(f"Logging in to {profile.database or 'the server'} as {profile.user}")
    session = Session(profile, d.connect(profile, password), d)
    if profile.dialect != "sqlite" and profile.ssl != "off" and not profile.tunneled:
        # the driver can be talked out of TLS by a server (or a man in the
        # middle) that claims not to support it; a profile that asked for
        # encryption gets it or gets nothing
        if d.encrypted(session) is False:
            session.close()
            raise DbError(_no_ssl(profile))
    return session

database_of

database_of(profile)

The database segment of a table address on this profile: main for a file, else whatever database the profile is pointed at.

Source code in src\taters\helpers\db.py
2539
2540
2541
2542
def database_of(profile: Profile) -> str:
    """The database segment of a table address on this profile: ``main``
    for a file, else whatever database the profile is pointed at."""
    return "main" if profile.dialect == "sqlite" else profile.database

drop_table

drop_table(session, entry)

Remove a table Taters made, and its registry and sidecar rows.

Source code in src\taters\helpers\db.py
2459
2460
2461
2462
2463
2464
2465
2466
def drop_table(session: Session, entry: Registered) -> None:
    """Remove a table Taters made, and its registry and sidecar rows."""
    d, schema = session.d, session.profile.schema
    d.drop(session, entry.table_name, schema)
    session.run(f"DELETE FROM {d.qualified(REGISTRY, schema)} WHERE table_name = ?",
                (entry.table_name,))
    session.run(f"DELETE FROM {d.qualified(SIDECARS, schema)} WHERE table_name = ?",
                (entry.table_name,))

fingerprint_table

fingerprint_table(session, entry)

Recompute a written table's fingerprint from its rows.

Source code in src\taters\helpers\db.py
2451
2452
2453
2454
2455
2456
def fingerprint_table(session: Session, entry: Registered) -> str:
    """Recompute a written table's fingerprint from its rows."""
    fp = Fingerprint(entry.logical_columns)
    for cells in read_cells(session, entry):
        fp.add(cells)
    return fp.hexdigest()

for_database

for_database(profile, database='')

The profile pointed at one database on its server. main and "" mean the file itself for SQLite; "" means the saved default otherwise.

Source code in src\taters\helpers\db.py
2529
2530
2531
2532
2533
2534
2535
2536
def for_database(profile: Profile, database: str = "") -> Profile:
    """The profile pointed at one database on its server. ``main`` and ""
    mean the file itself for SQLite; "" means the saved default otherwise."""
    if profile.dialect == "sqlite":
        return profile if profile.database == "main" else with_fields(profile, database="main")
    if not database or database == profile.database:
        return profile
    return with_fields(profile, database=database)

host_keys

host_keys(profile, log='')

The keys the ssh host presents, by asking it (ssh-keyscan), or from what plink already printed when it refused: its complaint names the key's type and fingerprint. Empty when neither is to be had.

Source code in src\taters\helpers\db.py
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
def host_keys(profile: Profile, log: str = "") -> List[HostKey]:
    """
    The keys the ssh host presents, by asking it (``ssh-keyscan``), or from
    what plink already printed when it refused: its complaint names the
    key's type and fingerprint. Empty when neither is to be had.
    """
    import shutil
    import subprocess

    found: List[HostKey] = []
    port = profile.ssh_port or 22
    if shutil.which("ssh-keyscan"):
        try:
            out = subprocess.run(["ssh-keyscan", "-p", str(port), "-T", "10", profile.ssh_host],
                                 capture_output=True, text=True, timeout=20).stdout
        except (OSError, subprocess.SubprocessError):
            out = ""
        for line in out.splitlines():
            parts = line.split()
            if len(parts) >= 3 and not line.startswith("#"):
                try:
                    found.append(HostKey(parts[1], parts[2], _fingerprint(parts[2])))
                except Exception:
                    continue
    if not found and not uses_putty(profile) and shutil.which("ssh"):
        # no ssh-keyscan (or it got nothing): ssh itself will write the key
        # it is shown into a known_hosts file of our choosing, and a
        # throwaway file is one nothing trusts. the login is expected to
        # fail; the file is what was wanted
        found = _keys_via_ssh(profile)
    if not found:
        # plink: "ssh-ed25519 255 SHA256:AbC..." somewhere in its refusal
        m = re.search(r"((?:ssh|ecdsa)-[\w@.-]+)\s+\d+\s+(SHA256:[A-Za-z0-9+/=]+)", log)
        if m:
            found.append(HostKey(m.group(1), "", m.group(2)))
    return found

list_profiles

list_profiles()

Every saved connection, by name. A damaged entry is skipped.

Source code in src\taters\helpers\db.py
260
261
262
263
264
265
266
267
268
269
def list_profiles() -> List[Profile]:
    """Every saved connection, by name. A damaged entry is skipped."""
    doc = _settings.load_settings().get(CONNECTIONS_KEY) or {}
    out = []
    for name, entry in sorted(doc.items()):
        if isinstance(entry, Mapping):
            p = _profile_from(str(name), entry)
            if p is not None:
                out.append(p)
    return out

login_by_hand

login_by_hand(profile)

Run the by-hand login right here, in this terminal, with nothing captured: ssh (or plink) shows the server's key and asks whether to trust it, exactly as it would at a shell prompt, and remembers the answer where it always does. For when the key could not be fetched to show inside the app. Returns the program's exit code.

Source code in src\taters\helpers\db.py
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
def login_by_hand(profile: Profile) -> int:
    """
    Run the by-hand login right here, in this terminal, with nothing
    captured: ssh (or plink) shows the server's key and asks whether to
    trust it, exactly as it would at a shell prompt, and remembers the
    answer where it always does. For when the key could not be fetched to
    show inside the app. Returns the program's exit code.
    """
    import subprocess

    who = (f"{profile.ssh_user}@" if profile.ssh_user else "") + profile.ssh_host
    key = str(Path(profile.ssh_key).expanduser()) if profile.ssh_key else ""
    if uses_putty(profile):
        cmd = [plink_path() or "plink", "-ssh"]
        if profile.ssh_port and profile.ssh_port != 22:
            cmd += ["-P", str(profile.ssh_port)]
        cmd += ["-i", key, who, "exit"]
    else:
        cmd = ["ssh"]
        if profile.ssh_port and profile.ssh_port != 22:
            cmd += ["-p", str(profile.ssh_port)]
        if key:
            cmd += ["-i", key]
        cmd += [who, "exit"]
    try:
        return subprocess.run(cmd).returncode
    except OSError:
        return 255

open_tunnel

open_tunnel(profile, timeout=TUNNEL_TIMEOUT)

The local port the database is reachable on through the SSH host, opening the tunnel on first use. A tunnel that cannot be opened says why in ssh's own words, which name the key, the host or the permission that was the problem; one that stalls says how far it got.

Source code in src\taters\helpers\db.py
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
def open_tunnel(profile: Profile, timeout: float = TUNNEL_TIMEOUT) -> int:
    """
    The local port the database is reachable on through the SSH host,
    opening the tunnel on first use. A tunnel that cannot be opened says
    why in ssh's own words, which name the key, the host or the permission
    that was the problem; one that stalls says how far it got.
    """
    import shutil
    import subprocess
    import threading
    import time
    from collections import deque

    have = _TUNNELS.get(profile.name)
    if have is not None and have[0].poll() is None and _port_open(have[1]):
        return have[1]
    if have is not None:
        _TUNNELS.pop(profile.name, None)
    if uses_putty(profile):
        if plink_path() is None:
            raise DbError(f"{profile.name} logs in with a PuTTY key ({Path(profile.ssh_key).name}), "
                          f"which needs PuTTY's plink, and it was not found on this computer. "
                          f"Install PuTTY (plink comes with it), or convert the key to "
                          f"OpenSSH's format in PuTTYgen (Conversions > Export OpenSSH key) "
                          f"and pick that file instead.")
    elif shutil.which("ssh") is None:
        raise DbError(f"{profile.name} is reached through ssh, and no ssh program was "
                      f"found on this computer. Install OpenSSH (Windows: Settings > "
                      f"Apps > Optional features > OpenSSH Client) and try again.")
    port = _free_port()
    proc = subprocess.Popen(tunnel_command(profile, port), stdin=subprocess.DEVNULL,
                            stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
    # what ssh says, read as it says it: a full pipe would stall ssh itself,
    # and the last lines are the story when nothing else is
    said: deque = deque(maxlen=200)

    def collect() -> None:
        for raw in iter(proc.stderr.readline, b""):
            said.append(raw.decode("utf-8", "replace").rstrip())
    threading.Thread(target=collect, daemon=True).start()
    deadline = time.monotonic() + timeout
    while time.monotonic() < deadline:
        if proc.poll() is not None:
            time.sleep(0.2)                    # let the reader drain the tail
            log = "\n".join(said)
            if _host_key_unknown(log):
                # the one refusal a person can settle on the spot: here is
                # the key the server showed, do you accept it
                raise HostKeyUnknown(_ssh_words(log, profile), profile,
                                     keys=host_keys(profile, log))
            raise DbError(_ssh_words(log, profile))
        if _port_open(port):
            _TUNNELS[profile.name] = (proc, port)
            return port
        time.sleep(0.2)
    proc.kill()
    raise DbError(_stall_words("\n".join(said), profile, timeout))

parse_login_hints

parse_login_hints(text, profile)

The look's output as :class:LoginHints; separated out so it can be tested without a server.

Source code in src\taters\helpers\db.py
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
def parse_login_hints(text: str, profile: Profile) -> LoginHints:
    """The look's output as :class:`LoginHints`; separated out so it can be
    tested without a server."""
    sections: Dict[str, List[str]] = {}
    current = ""
    for line in text.splitlines():
        if line.strip() in ("--mycnf--", "--pgpass--", "--sockets--", "--user--"):
            current = line.strip().strip("-")
            sections[current] = []
        elif current:
            sections[current].append(line)
    os_user = next((ln.strip() for ln in sections.get("user", []) if ln.strip()), "")
    sockets = dict(ln.split("=", 1) for ln in sections.get("sockets", []) if "=" in ln)
    socket = sockets.get(profile.dialect, "")
    user, password, source = "", None, ""
    if profile.dialect == "mysql":
        client = _my_cnf_client("\n".join(sections.get("mycnf", [])))
        user = client.get("user", "")
        password = client.get("password")
        socket = client.get("socket", socket) or socket
        source = "~/.my.cnf" if client else ""
    elif profile.dialect == "postgres":
        hit = _pgpass_match("\n".join(sections.get("pgpass", [])), profile)
        if hit:
            user, password, source = hit[0], hit[1], "~/.pgpass"
    return LoginHints(user=user, password=password, socket=socket, source=source,
                      os_user=os_user)

password_env

password_env(name)

lab-db -> TATERS_DB_PASSWORD_LAB_DB.

Source code in src\taters\helpers\db.py
246
247
248
def password_env(name: str) -> str:
    """``lab-db`` -> ``TATERS_DB_PASSWORD_LAB_DB``."""
    return "TATERS_DB_PASSWORD_" + re.sub(r"[^A-Z0-9]", "_", name.upper())

password_for

password_for(profile, ask=None)

The password for a profile, from the environment, then this session's memory, then ask(question) -- or None when the profile needs none.

ask is the masked prompt; without one (a script, a worker), a password that is not already known raises and names the variable to set, so a headless run fails at once rather than at step seven.

Source code in src\taters\helpers\db.py
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
def password_for(profile: Profile,
                 ask: Optional[Callable[[str], str]] = None) -> Optional[str]:
    """
    The password for a profile, from the environment, then this session's
    memory, then ``ask(question)`` -- or None when the profile needs none.

    ``ask`` is the masked prompt; without one (a script, a worker), a
    password that is not already known raises and names the variable to
    set, so a headless run fails at once rather than at step seven.
    """
    if profile.dialect == "sqlite" or profile.password == "none":
        return None
    env = os.environ.get(password_env(profile.name))
    if env:
        return env
    if profile.name in _SECRETS:
        return _SECRETS[profile.name]
    if profile.password == "server":
        # the person's own saved login on the ssh host, read over the same
        # ssh login each session and kept only in memory, like a typed one
        hints = remote_login_hints(profile)
        if hints.password is None:
            raise NotPermitted(
                f"{profile.name} is set to take its password from your saved login on "
                f"{profile.ssh_host}, and {hints.problem or 'none was found there'}. Set "
                f"the connection to ask instead (Settings > Database connections).")
        _SECRETS[profile.name] = hints.password
        return hints.password
    if ask is None:
        raise NotPermitted(
            f"{profile.name} needs a password and none is known. Set "
            f"{password_env(profile.name)} in the environment, or run the "
            f"app, which asks for it.")
    value = ask(f"Password for {profile.user} on {profile.name}:")
    _SECRETS[profile.name] = value
    return value
plink_path()

Where plink is: on the PATH, or in PuTTY's usual Windows folders.

Source code in src\taters\helpers\db.py
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
def plink_path() -> Optional[str]:
    """Where plink is: on the PATH, or in PuTTY's usual Windows folders."""
    import shutil

    found = shutil.which("plink")
    if found:
        return found
    for folder in (os.environ.get("ProgramFiles", r"C:\Program Files"),
                   os.environ.get("ProgramFiles(x86)", r"C:\Program Files (x86)")):
        candidate = Path(folder) / "PuTTY" / "plink.exe"
        if candidate.is_file():
            return str(candidate)
    return None

probe

probe(profile, password=None, ask=None, on_step=None)

Try each thing a run needs and say which worked, by name. on_step hears each thing as it is tried, so nothing here is a silent wait.

Source code in src\taters\helpers\db.py
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
def probe(profile: Profile, password: Optional[str] = None,
          ask: Optional[Callable[[str], str]] = None,
          on_step: Optional[Callable[[str], None]] = None) -> Probe:
    """Try each thing a run needs and say which worked, by name. ``on_step``
    hears each thing as it is tried, so nothing here is a silent wait."""
    notes: List[str] = []
    say = on_step or (lambda text: None)
    try:
        session = connect(profile, password, ask, on_step=on_step)
    except HostKeyUnknown:
        raise                       # the one refusal a person can settle on the spot
    except DbError as e:
        return Probe(notes=(str(e),))
    with session:
        d = session.d
        try:
            say("Asking the server its version")
            session.one("SELECT 1")
            version = d.version(session)
            say("Counting the tables this account can see")
            tables = d.count_tables(session, profile.schema)
            encrypted = d.encrypted(session)
        except Exception as e:
            return Probe(notes=(f"{type(e).__name__}: {e}",))
        name = f"_taters_probe_{_secrets.token_hex(4)}"
        create = drop = False
        try:
            say("Checking whether this account can create a table")
            d.create(session, name, [("x", d.text_type)], schema=profile.schema)
            create = True
        except Exception as e:
            notes.append(f"create: {_short(e)}")
        if create:
            try:
                say("...and remove it again")
                d.drop(session, name, profile.schema)
                drop = True
            except Exception as e:
                notes.append(f"drop: {_short(e)}")
        registry = False
        if create:
            try:
                say("Setting up the small registry Taters keeps of its tables")
                ensure_registry(session)
                registry = True
            except Exception as e:
                notes.append(f"registry: {_short(e)}")
    return Probe(select=True, create=create, drop=drop, registry=registry,
                 version=version, tables=tables, encrypted=encrypted,
                 notes=tuple(notes))

query_words

query_words(e, p)

What the server said while a query ran, in words a person can act on. The login explainers cover not getting in; this covers the things that go wrong once in -- the server's disk filling up, the connection dropping, a table the account may not read -- which are almost always somebody else's to fix, and the sentence says whose.

Source code in src\taters\helpers\db.py
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
def query_words(e: Exception, p: Profile) -> str:
    """
    What the server said while a query ran, in words a person can act on.
    The login explainers cover not getting in; this covers the things that
    go wrong once in -- the server's disk filling up, the connection
    dropping, a table the account may not read -- which are almost always
    somebody else's to fix, and the sentence says whose.
    """
    text = str(e)
    low = text.lower()
    code = e.args[0] if getattr(e, "args", None) and isinstance(e.args[0], int) else None
    ask = ("Ask whoever runs the database server" if p.dialect != "sqlite"
           else "Free some space on this computer")
    if (code in (1021, 1114) or "errno: 28" in low or "no space left" in low
            or "disk is full" in low or "disk full" in low):
        return (f"{p.engine_label} on {p.name} has run out of disk space"
                + (" (its temp space, which it needs to answer a query)"
                   if code == 1021 or "/tmp" in text else "")
                + f". Nothing is wrong on this computer. {ask}; until then "
                f"the database cannot answer.")
    if code in (2013, 2006) or "server has gone away" in low or \
            "closed the connection unexpectedly" in low or "connection was killed" in low:
        return (f"The connection to {p.name} dropped mid-query"
                + (" (the ssh tunnel may have closed)" if p.tunneled else "")
                + ". Try again; if it keeps happening on one table, the server may "
                "be killing long queries.")
    if code in (1142, 1143, 1044, 1045) or "permission denied" in low or "access denied" in low \
            or "command denied" in low:
        return (f"{p.user or 'This account'} is not allowed to do that on {p.name}: "
                f"{_short(e)}. Ask whoever runs the database for the right.")
    if code in (1205, 1213) or "lock wait timeout" in low or "deadlock" in low \
            or "database is locked" in low:
        return (f"{p.name} is busy: another session holds a lock on what this needs. "
                f"Try again in a moment.")
    if code in (1146,) or ("no such table" in low) or ("does not exist" in low and
                                                       "relation" in low):
        return f"{p.name} has no such table any more: {_short(e)}."
    return f"{p.engine_label} on {p.name} said: {_short(e)}"

read_cells

read_cells(session, entry, *, columns=None, on_rows=None)

The rows of a table Taters wrote, in write order, as lists of text cells in columns order (the table's own when none are named).

Source code in src\taters\helpers\db.py
2400
2401
2402
2403
2404
2405
2406
2407
2408
2409
2410
2411
2412
2413
2414
2415
2416
2417
2418
2419
2420
2421
2422
2423
2424
2425
2426
2427
2428
2429
2430
2431
2432
2433
2434
2435
2436
2437
2438
2439
def read_cells(session: Session, entry: Registered, *,
               columns: Optional[Sequence[str]] = None,
               on_rows: Optional[Callable[[int], None]] = None) -> Iterator[List[str]]:
    """The rows of a table Taters wrote, in write order, as lists of text
    cells in ``columns`` order (the table's own when none are named)."""
    d, schema = session.d, session.profile.schema
    wanted = [str(c) for c in columns] if columns else entry.logical_columns
    logical = entry.logical_columns
    missing = [c for c in wanted if c not in logical]
    if missing:
        raise DbError(f"{entry.logical} has no column {missing[0]!r}")
    phys = {c[0]: c[1] for c in entry.columns if c[1]}
    packed = entry.layout == "packed"
    packed_order = [c[0] for c in entry.columns if not c[1]] if packed else []
    # select what the table really has; the packed column is unpacked here
    # and the wanted cells picked out, so a reader never sees the layout
    real_wanted = [c for c in wanted if c in phys]
    need_packed = packed and any(c not in phys for c in wanted)
    selected = [d.quote(ROW_COLUMN), *(d.quote(phys[c]) for c in real_wanted)]
    if need_packed:
        selected.append(d.quote(PACKED_COLUMN))
    sql = (f"SELECT {', '.join(selected)} FROM {d.qualified(entry.table_name, schema)} "
           f"WHERE {d.quote(ROW_COLUMN)} > ? ORDER BY {d.quote(ROW_COLUMN)}")
    last = 0
    n = 0
    while True:
        rows = session.all(d.limit(sql, _BATCH), (last,))
        if not rows:
            break
        for row in rows:
            last = int(row[0])
            n += 1
            have = dict(zip(real_wanted, (cell_text(v) for v in row[1:1 + len(real_wanted)])))
            if need_packed:
                have.update(zip(packed_order, _unpack(row[-1], len(packed_order))))
            yield [have.get(c, "") for c in wanted]
        if on_rows is not None:
            on_rows(n)
    if on_rows is not None:
        on_rows(n)

read_rows

read_rows(session, entry, *, columns=None, on_rows=None)

The rows of a table Taters wrote, in write order, as dicts of text cells.

Source code in src\taters\helpers\db.py
2442
2443
2444
2445
2446
2447
2448
def read_rows(session: Session, entry: Registered, *,
              columns: Optional[Sequence[str]] = None,
              on_rows: Optional[Callable[[int], None]] = None) -> Iterator[Dict[str, str]]:
    """The rows of a table Taters wrote, in write order, as dicts of text cells."""
    wanted = [str(c) for c in columns] if columns else entry.logical_columns
    for cells in read_cells(session, entry, columns=wanted, on_rows=on_rows):
        yield dict(zip(wanted, cells))

registered_under

registered_under(session, prefix)

Every registered table whose logical path starts with prefix.

Source code in src\taters\helpers\db.py
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
def registered_under(session: Session, prefix: str) -> List[Registered]:
    """Every registered table whose logical path starts with ``prefix``."""
    d, schema = session.d, session.profile.schema
    if not d.table_exists(session, REGISTRY, schema):
        return []
    like = prefix.rstrip("/") + "/%"
    rows = session.all(f"SELECT table_name FROM {d.qualified(REGISTRY, schema)} WHERE "
                       f"logical LIKE ? ORDER BY logical", (like,))
    out = []
    for (name,) in rows:
        r = registered(session, str(name))
        if r:
            out.append(r)
    return out

remote_command

remote_command(profile, command, timeout=30.0)

Run one shell command on the ssh host, in batch mode, through the same program and key the tunnel uses. (exit code, stdout, stderr).

Source code in src\taters\helpers\db.py
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
def remote_command(profile: Profile, command: str, timeout: float = 30.0) -> Tuple[int, str, str]:
    """Run one shell command on the ssh host, in batch mode, through the
    same program and key the tunnel uses. ``(exit code, stdout, stderr)``."""
    import subprocess

    who = (f"{profile.ssh_user}@" if profile.ssh_user else "") + profile.ssh_host
    key = str(Path(profile.ssh_key).expanduser()) if profile.ssh_key else ""
    if uses_putty(profile):
        cmd = [plink_path() or "plink", "-ssh", "-batch"]
        if profile.ssh_port and profile.ssh_port != 22:
            cmd += ["-P", str(profile.ssh_port)]
        for fp in [f.strip() for f in profile.ssh_hostkey.split(",") if f.strip()]:
            cmd += ["-hostkey", fp]
        cmd += ["-i", key, who, command]
    else:
        cmd = ["ssh", "-o", "BatchMode=yes"]
        if profile.ssh_port and profile.ssh_port != 22:
            cmd += ["-p", str(profile.ssh_port)]
        if key:
            cmd += ["-i", key]
        cmd += [who, command]
    done = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout,
                          stdin=subprocess.DEVNULL)
    return done.returncode, done.stdout, done.stderr

remote_login_hints

remote_login_hints(profile)

Look on the ssh host for how its own database client logs in, with the person's consent already given: their ~/.my.cnf (MySQL) or ~/.pgpass (PostgreSQL), and whether the database's socket is there. A failure to log in comes back as problem, never as an exception, because this is a look, not the connection.

Source code in src\taters\helpers\db.py
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
def remote_login_hints(profile: Profile) -> LoginHints:
    """
    Look on the ssh host for how its own database client logs in, with the
    person's consent already given: their ``~/.my.cnf`` (MySQL) or
    ``~/.pgpass`` (PostgreSQL), and whether the database's socket is there.
    A failure to log in comes back as ``problem``, never as an exception,
    because this is a look, not the connection.
    """
    import subprocess

    try:
        code, out, err = remote_command(profile, _LOOK)
    except (OSError, subprocess.SubprocessError) as e:
        return LoginHints(problem=f"could not run a command on {profile.ssh_host}: {e}")
    if code != 0 and "--mycnf--" not in out:
        if _host_key_unknown(err):
            raise HostKeyUnknown(_ssh_words(err, profile), profile, keys=host_keys(profile, err))
        return LoginHints(problem=_ssh_words(err, profile))
    return parse_login_hints(out, profile)

sanitize_table_name

sanitize_table_name(logical)

taters/study7/features/readability -> taters__study7__features__readability.

Lowercase letters, digits and underscores; past 48 characters the name is shortened deterministically with a hash, never by chance.

Source code in src\taters\helpers\db.py
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
def sanitize_table_name(logical: str) -> str:
    """
    ``taters/study7/features/readability`` -> ``taters__study7__features__readability``.

    Lowercase letters, digits and underscores; past 48 characters the name
    is shortened deterministically with a hash, never by chance.
    """
    parts = [p for p in str(logical).replace("\\", "/").split("/") if p]
    clean = []
    for p in parts:
        s = re.sub(r"[^a-z0-9_]+", "_", p.lower()).strip("_") or "t"
        if s[0].isdigit():
            s = "t_" + s
        clean.append(s)
    name = "__".join(clean) or "t"
    if len(name) > _NAME_LIMIT:
        name = name[:_NAME_LIMIT - 9] + "_" + hashlib.sha1(logical.encode("utf-8")).hexdigest()[:8]
    return name

save_profile

save_profile(profile)

Write a profile, keeping the others. Never a password.

Source code in src\taters\helpers\db.py
280
281
282
283
284
285
def save_profile(profile: Profile) -> Path:
    """Write a profile, keeping the others. Never a password."""
    doc = dict(_settings.load_settings().get(CONNECTIONS_KEY) or {})
    entry = profile.to_dict()
    doc[profile.name] = entry
    return _settings.save_setting(CONNECTIONS_KEY, doc)

session_for

session_for(name, database='', ask=None)

A session by connection name, on one of its databases; the usual entry from the table layer. A connection is a server and a login; which database to use on it is chosen each time, so it is passed here. Empty means the connection's own default (a SQLite file has only the one).

Source code in src\taters\helpers\db.py
2517
2518
2519
2520
2521
2522
2523
2524
2525
2526
def session_for(name: str, database: str = "",
                ask: Optional[Callable[[str], str]] = None) -> Session:
    """
    A session by connection name, on one of its databases; the usual entry
    from the table layer. A connection is a server and a login; which
    database to use on it is chosen each time, so it is passed here. Empty
    means the connection's own default (a SQLite file has only the one).
    """
    profile = get_profile(name)
    return connect(for_database(profile, database), ask=ask)

ssh_hint

ssh_hint(p)

The command a person would type to log in to this host by hand: ssh -p 2200 me@host, or with PuTTY plink -P 2200 -i key.ppk me@host -- the port goes on a flag, not after a colon.

Source code in src\taters\helpers\db.py
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
def ssh_hint(p: Profile) -> str:
    """The command a person would type to log in to this host by hand:
    ``ssh -p 2200 me@host``, or with PuTTY ``plink -P 2200 -i key.ppk me@host``
    -- the port goes on a flag, not after a colon."""
    who = (f"{p.ssh_user}@" if p.ssh_user else "") + p.ssh_host
    if uses_putty(p):
        port = f"-P {p.ssh_port} " if p.ssh_port and p.ssh_port != 22 else ""
        return f"plink {port}-i {Path(p.ssh_key).name} {who}"
    port = f"-p {p.ssh_port} " if p.ssh_port and p.ssh_port != 22 else ""
    return f"ssh {port}{who}"

stream_query

stream_query(session, sql, params=(), *, on_rows=None)

A query's column names and its rows as text cells, streamed.

Source code in src\taters\helpers\db.py
2499
2500
2501
2502
2503
2504
2505
2506
2507
2508
2509
2510
2511
2512
2513
2514
def stream_query(session: Session, sql: str, params: Sequence[Any] = (), *,
                 on_rows: Optional[Callable[[int], None]] = None
                 ) -> Tuple[List[str], Iterator[Dict[str, str]]]:
    """A query's column names and its rows as text cells, streamed."""
    columns = session.columns_of_query(sql, params)

    def rows() -> Iterator[Dict[str, str]]:
        n = 0
        for row in session.stream(sql, params):
            n += 1
            yield {c: cell_text(v) for c, v in zip(columns, row)}
            if on_rows is not None and n % _BATCH == 0:
                on_rows(n)
        if on_rows is not None:
            on_rows(n)
    return columns, rows()

tunnel_command

tunnel_command(profile, local_port)

The invocation that forwards local_port to the database as the SSH host sees it: OpenSSH's ssh, or PuTTY's plink for a .ppk key. Batch mode either way -- a key, an agent or Pageant, never a prompt, since there is no terminal to answer one in the middle of a run.

Source code in src\taters\helpers\db.py
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
def tunnel_command(profile: Profile, local_port: int) -> List[str]:
    """
    The invocation that forwards ``local_port`` to the database as the SSH
    host sees it: OpenSSH's ``ssh``, or PuTTY's ``plink`` for a ``.ppk``
    key. Batch mode either way -- a key, an agent or Pageant, never a
    prompt, since there is no terminal to answer one in the middle of a run.
    """
    # a socket on the far end instead of a port: the server then sees the
    # ssh user, not a network client (OpenSSH only; plink cannot forward one)
    far = profile.socket if profile.socket else \
        f"{profile.host or 'localhost'}:{profile.port or DEFAULT_PORT[profile.dialect]}"
    target = f"127.0.0.1:{local_port}:{far}"
    who = (f"{profile.ssh_user}@" if profile.ssh_user else "") + profile.ssh_host
    key = str(Path(profile.ssh_key).expanduser()) if profile.ssh_key else ""
    if uses_putty(profile):
        if profile.socket:
            raise DbError(f"{profile.name} logs in over the database's socket on "
                          f"{profile.ssh_host}, which PuTTY's plink cannot forward. Use an "
                          f"OpenSSH key for this connection (PuTTYgen: Conversions > Export "
                          f"OpenSSH key), or give it the database's port and a password.")
        cmd = [plink_path() or "plink", "-ssh", "-batch", "-v", "-N", "-L", target]
        if profile.ssh_port and profile.ssh_port != 22:
            cmd += ["-P", str(profile.ssh_port)]
        for fp in [f.strip() for f in profile.ssh_hostkey.split(",") if f.strip()]:
            cmd += ["-hostkey", fp]
        cmd += ["-i", key, who]
        return cmd
    cmd = ["ssh", "-v", "-N", "-o", "BatchMode=yes", "-o", "ExitOnForwardFailure=yes",
           "-o", "ServerAliveInterval=30", "-L", target]
    if profile.ssh_port and profile.ssh_port != 22:
        cmd += ["-p", str(profile.ssh_port)]
    if key:
        cmd += ["-i", key]
    cmd.append(who)
    return cmd

uses_putty

uses_putty(profile)

A .ppk key is PuTTY's format, which OpenSSH cannot read; the tunnel is then opened with PuTTY's own command-line client, plink, which also uses Pageant and the host keys PuTTY has already accepted.

Source code in src\taters\helpers\db.py
1027
1028
1029
1030
1031
def uses_putty(profile: Profile) -> bool:
    """A ``.ppk`` key is PuTTY's format, which OpenSSH cannot read; the
    tunnel is then opened with PuTTY's own command-line client, plink,
    which also uses Pageant and the host keys PuTTY has already accepted."""
    return profile.ssh_key.lower().endswith(".ppk")

write_rows

write_rows(
    session,
    logical,
    columns,
    rows,
    *,
    hints=None,
    on_rows=None
)

Load rows into the table named by logical; see :class:Loader.

Source code in src\taters\helpers\db.py
2344
2345
2346
2347
2348
2349
2350
2351
2352
2353
2354
2355
2356
def write_rows(session: Session, logical: str, columns: Sequence[str],
               rows: Iterable[Sequence[Any]], *,
               hints: Optional[Mapping[str, str]] = None,
               on_rows: Optional[Callable[[int], None]] = None) -> Registered:
    """Load ``rows`` into the table named by ``logical``; see :class:`Loader`."""
    loader = Loader(session, logical, columns, hints=hints, on_rows=on_rows)
    try:
        for row in rows:
            loader.add(row)
        return loader.finish()
    except BaseException:
        loader.abort()
        raise

taters.helpers.feature_average

Average a feature table up to a coarser row.

The other way round from :mod:text_gather. That one joins the texts that belong together and measures the result once; this one measures every row on its own and then averages the numbers. They are different analyses, not two routes to one answer -- joining a speaker's words before measuring versus averaging their per-utterance scores differ by around 35% on vocabulary measures -- so which one happened is the user's choice and is recorded.

One table in, one table out, because a feature table's record has to sit beside it: :func:taters.stats.assemble.assemble_analysis_table reads the record of every table it joins to work out which columns are bookkeeping, and :mod:taters.score_model reads it again to check a saved model against the features it was fitted on. A step that wrote a folder full of tables would leave all of them unrecorded.

Averaging repeats. Utterances to speakers to conversations is two calls, the second reading the first's output, and it is not the same as one call straight to conversations: the two-step version weights speakers equally, the one-step version weights whoever talked most. That is the reason this takes a table rather than a corpus -- so the chain is visible in the pipeline instead of hidden in an argument.

Nothing here renames a column. A model is fitted on column names and has to be applicable to text that arrives at any shape, so flesch_reading_ease stays flesch_reading_ease whether it describes one utterance or a conversation's mean of means. The pipeline and its manifest are what record the difference; see the defines_text note in :mod:taters.helpers.provenance.

average_feature_table

average_feature_table(
    *,
    in_csv,
    out_csv,
    group_by,
    keys_csv=None,
    key_cols=("text_id",),
    split_col="",
    min_rows=1,
    overwrite_existing=False,
    delimiter=",",
    encoding="utf-8-sig",
    on_progress=None,
    verbose=True
)

One row per group, each number the mean of the rows that went into it.

Parameters:

Name Type Description Default
in_csv str or Path

The feature table to average: one row per text, a row key, and numeric columns. Non-numeric columns are dropped, since there is no mean of a word. A file, or db://<connection>/<database>/<table>.

required
out_csv str or Path

Where to write the averaged table: a file, or a database table.

required
group_by sequence of str

The columns whose shared values make a group. They come from keys_csv on the first average and from in_csv itself on any later one.

required
keys_csv str or Path

A table saying which row belongs to which group: the row key, and the grouping columns. Needed for the first average of a feature table, whose only non-numeric column is its key. Omit it once the input already carries the grouping columns.

None
key_cols sequence of str

What identifies a row in both tables, and the name the averaged row key is written under. The averaged key is the group's values joined with " | ", exactly as the text gather composes a grouped id, so the two tables join.

``("text_id",)``
split_col str

A column that splits a row legitimately and must survive the average rather than be grouped away -- source_col when each text column was measured separately. Added to the grouping when present.

""
min_rows int

Leave out groups assembled from fewer rows than this. 1 keeps everything.

1
overwrite_existing bool

Rebuild the table if it is already there.

False
delimiter str

How to read the CSVs. The output is written comma-separated, as every feature table is.

','
encoding str

How to read the CSVs. The output is written comma-separated, as every feature table is.

','
on_progress callable

Progress callback.

None
verbose bool

Print a line about what was averaged.

True

Returns:

Type Description
Path or TableRef

out_csv: the path for a file, the ref for a database table.

Notes

A group whose values are all blank gets the row key "group", matching :func:taters.helpers.text_gather.csv_to_analysis_ready_csv. A blank cell is skipped rather than counted as zero, so a column measured on only some of a speaker's turns is the mean of the turns it was measured on; the count column reports the rows in the group, not the rows behind any one number.

Source code in src\taters\helpers\feature_average.py
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
@records_settings(
    # which table was read, and what it was matched against. recorded for
    # diagnosis, never compared: a model exists to meet a different corpus.
    binding=("in_csv", "keys_csv", "key_cols", "delimiter", "encoding"),
    grain=_GRAIN,
    outputs=("out_csv",),
    # averaging is not measuring: the record takes the identity of whatever
    # measured `in_csv`, so a model fitted on averages can be scored on the
    # un-averaged table, and the measure's own bookkeeping columns are not
    # forgotten on the way up.
    reshapes="in_csv",
    # cheap enough that an existing copy with no record gets redone rather
    # than trusted -- same reasoning as the two text gathers.
    redo_without_record=True,
    bookkeeping=(COUNT_COLUMN,),
    # this step decides what a row *is*, not how anything was measured. its
    # settings stay out of the chain downstream steps compare, which is what
    # lets a model fitted on averages score un-averaged text. the module
    # docstring in `provenance` names this case outright.
    defines_text=True)
def average_feature_table(
    *,
    in_csv: PathLike,
    out_csv: PathLike,
    group_by: Sequence[str],
    keys_csv: Optional[PathLike] = None,
    key_cols: Sequence[str] = ("text_id",),
    split_col: str = "",
    min_rows: int = 1,
    overwrite_existing: bool = False,
    delimiter: str = ",",
    encoding: str = "utf-8-sig",
    on_progress: Optional[Callable[..., None]] = None,
    verbose: bool = True,
) -> Path:
    """
    One row per group, each number the mean of the rows that went into it.

    Parameters
    ----------
    in_csv : str or Path
        The feature table to average: one row per text, a row key, and numeric
        columns. Non-numeric columns are dropped, since there is no mean of a
        word. A file, or ``db://<connection>/<database>/<table>``.
    out_csv : str or Path
        Where to write the averaged table: a file, or a database table.
    group_by : sequence of str
        The columns whose shared values make a group. They come from
        ``keys_csv`` on the first average and from ``in_csv`` itself on any
        later one.
    keys_csv : str or Path, optional
        A table saying which row belongs to which group: the row key, and the
        grouping columns. Needed for the first average of a feature table,
        whose only non-numeric column is its key. Omit it once the input
        already carries the grouping columns.
    key_cols : sequence of str, default ``("text_id",)``
        What identifies a row in both tables, and the name the averaged row
        key is written under. The averaged key is the group's values joined
        with ``" | "``, exactly as the text gather composes a grouped id, so
        the two tables join.
    split_col : str, default ""
        A column that splits a row legitimately and must survive the average
        rather than be grouped away -- ``source_col`` when each text column
        was measured separately. Added to the grouping when present.
    min_rows : int, default 1
        Leave out groups assembled from fewer rows than this. ``1`` keeps
        everything.
    overwrite_existing : bool, default False
        Rebuild the table if it is already there.
    delimiter, encoding : str
        How to read the CSVs. The output is written comma-separated, as every
        feature table is.
    on_progress : callable, optional
        Progress callback.
    verbose : bool, default True
        Print a line about what was averaged.

    Returns
    -------
    pathlib.Path or TableRef
        ``out_csv``: the path for a file, the ref for a database table.

    Notes
    -----
    A group whose values are all blank gets the row key ``"group"``, matching
    :func:`taters.helpers.text_gather.csv_to_analysis_ready_csv`. A blank cell
    is skipped rather than counted as zero, so a column measured on only some
    of a speaker's turns is the mean of the turns it was measured on; the
    count column reports the rows in the group, not the rows behind any one
    number.
    """
    src = tables.parse_ref(in_csv)
    out = tables.parse_ref(out_csv)
    keys = list(group_by) + ([split_col] if split_col else [])
    if not keys:
        raise ValueError(
            "group_by is empty, so there is nothing to average by. Averaging "
            "a table by nothing would copy it.")
    if tables.exists(out) and not overwrite_existing:
        if verbose:
            print(f"[average] exists, leaving alone: {out}")
        return tables.handle(out)
    if not tables.exists(src):
        raise FileNotFoundError(f"in_csv not found: {src}")

    announce(on_progress, f"averaging {src.name} by {', '.join(keys)}")
    header = tables.columns_of(src, encoding=encoding, delimiter=delimiter)
    if not header:
        raise ValueError(f"{src.name} has no header row.")

    lookup = None
    if keys_csv is not None:
        lookup = _key_lookup(keys_csv, key_cols=key_cols,
                             group_by=group_by, encoding=encoding,
                             delimiter=delimiter)
        missing = [c for c in key_cols if c not in header]
        if missing:
            raise ValueError(
                f"{src.name} has no column(s) {missing}, so its rows cannot "
                f"be matched to the ones in {tables.parse_ref(keys_csv).name}.")
    else:
        absent = [c for c in keys if c not in header]
        if absent:
            raise ValueError(
                f"{src.name} has no column(s) {absent} to group by. Averaging "
                f"a table by a column it does not carry needs keys_csv, which "
                f"says which row belongs to which group.")

    # numeric columns are whatever parses somewhere. a column of words gives
    # no number anywhere and is dropped; one that's numeric in some rows and
    # blank in others is kept and averaged over the rows that had a value.
    # the count from a previous average is not a measurement, so it is not
    # averaged again -- and leaving it in put the column in the header twice,
    # once as a mean of counts and once as this level's own count.
    skip = set(keys) | set(key_cols) | {COUNT_COLUMN}
    candidates = [c for c in header if c not in skip]
    # one pass, and what we keep is per group: a running sum and a count per
    # column. the table itself streams through, so a million utterances cost
    # what their speakers cost
    sums = {}
    counts = {}
    members = {}
    order: List[tuple] = []
    n_rows = 0
    ticker = Ticker(on_progress, tables.count(src, encoding=encoding))
    for row in tables.read_rows(src, encoding=encoding, delimiter=delimiter):
        n_rows += 1
        ticker.tick(message="averaging rows")
        if lookup is not None:
            found = lookup.get(tuple(str(row.get(c, "")) for c in key_cols))
            if found is None:
                # a row the key map has never heard of. skipping it beats
                # inventing a group for it, and the count column is what
                # shows the rows went missing.
                continue
            group = found + ((str(row.get(split_col, "")),) if split_col else ())
        else:
            group = tuple(str(row.get(c, "")) for c in keys)
        if group not in members:
            members[group] = 0
            sums[group] = {}
            counts[group] = {}
            order.append(group)
        members[group] += 1
        for col in candidates:
            value = _numeric(row.get(col, ""))
            if value is None:
                continue
            sums[group][col] = sums[group].get(col, 0.0) + value
            counts[group][col] = counts[group].get(col, 0) + 1

    kept_cols = [c for c in candidates if any(counts[g].get(c) for g in order)]
    if not kept_cols:
        raise ValueError(
            f"{src.name} has no numeric column to average -- only "
            f"{', '.join(header) or 'nothing'}.")

    survivors = [g for g in order if members[g] >= max(1, int(min_rows))]
    if not survivors:
        raise ValueError(
            f"every group in {src.name} was assembled from fewer than "
            f"{min_rows} row(s), so nothing is left to write.")

    id_col = list(key_cols)[0]
    out_header = [id_col, *keys, COUNT_COLUMN, *kept_cols]
    hints = {**{c: "text" for c in [id_col, *keys]}, COUNT_COLUMN: "int",
             **{c: "float" for c in kept_cols}}
    with tables.write_table(out, out_header, encoding="utf-8", hints=hints) as writer:
        for group in survivors:
            row_id = _compose_id(group) or GROUP_FALLBACK_ID
            values = []
            for col in kept_cols:
                n = counts[group].get(col, 0)
                values.append(f"{sums[group][col] / n:.10g}" if n else "")
            writer.writerow([row_id, *group, members[group], *values])

    if verbose:
        dropped = len(order) - len(survivors)
        tail = f", {dropped} group(s) under {min_rows} row(s) left out" \
            if dropped else ""
        print(f"[average] {src.name}: {n_rows} row(s) -> "
              f"{len(survivors)} group(s) by {', '.join(keys)}{tail}")
    return tables.handle(out)

taters.helpers.library

The library: user-imported assets that outlive any one project.

A "dictionary" here is the general term -- a content-coding dictionary and an archetype dictionary are different kinds, with different formats, consumed by different modules. The library stores each kind in its own folder under the user's home, so importing a dictionary once makes it available to every pipeline on the machine, wherever it is run from. That replaces the old convention of a dictionaries/ folder relative to the working directory, which silently failed for anyone who did not happen to have one.

No UI imports (the :mod:taters.helpers.gpu pattern): the wizard is one consumer, but the runner or a future GUI can read the same library. All the operations are deliberately file-level and boring -- an entry is its file, its name is the filename stem -- so a user can also just manage the folder by hand and nothing here will be surprised.

Adding a new kind -- pretrained classifier models are the expected next one -- is a :data:KINDS entry plus a library= line on the recipe that consumes it; the manager, the picker, and the empty-library contingency in the wizard all key off those two declarations.

LibraryCollision

LibraryCollision(existing)

Bases: Exception

An import would overwrite an entry that already exists.

Raised instead of overwriting so the UI can ask replace-or-rename; the existing path rides along as .existing.

Source code in src\taters\helpers\library.py
374
375
376
def __init__(self, existing: Path):
    self.existing = existing
    super().__init__(f"'{existing.stem}' is already in the library")

LibraryKind dataclass

LibraryKind(
    id,
    label,
    help,
    suffixes,
    deep_check=None,
    describe_entry=None,
)

One category of importable asset.

Attributes:

Name Type Description
id str

Stable identifier; also the folder name under the library.

label, help str

What menus call it, and one sentence including the accepted formats.

suffixes tuple of str

File extensions this kind accepts, lowercase with the dot.

asset_problem

asset_problem(path, kind=None)

Why this file cannot work as a library asset, or "" when it can.

The canonical check, shared by import (refuse the file while the user is holding it and can act) and by the analyzers (a file can still arrive by CLI path without ever passing through the library). Two layers:

  • cheap shape heuristics, always: an empty file is not a dictionary, and a .dic with no %...% category header is a bare word list -- the mistake that made contentcoder die with a bare "list index out of range" mid-run, an hour after it was made;
  • with a kind, that kind's :attr:~LibraryKind.deep_check -- the real parser. Import passes the kind; the analyzers do not, because they construct the real parser on the very next line and asking it twice buys nothing.
Source code in src\taters\helpers\library.py
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
def asset_problem(path: Path, kind: Optional[LibraryKind] = None) -> str:
    """
    Why this file cannot work as a library asset, or "" when it can.

    The canonical check, shared by import (refuse the file while the user is
    holding it and can act) and by the analyzers (a file can still arrive by
    CLI path without ever passing through the library). Two layers:

    * cheap shape heuristics, always: an empty file is not a dictionary, and a
      ``.dic`` with no ``%...%`` category header is a bare word list -- the
      mistake that made contentcoder die with a bare "list index out of range"
      mid-run, an hour after it was made;
    * with a ``kind``, that kind's :attr:`~LibraryKind.deep_check` -- the real
      parser. Import passes the kind; the analyzers do not, because they
      construct the real parser on the very next line and asking it twice
      buys nothing.
    """
    path = Path(path)
    try:
        text = path.read_text(encoding="utf-8-sig", errors="ignore")
    except OSError as e:
        return f"'{path.name}' could not be read: {e}"
    if not text.strip():
        return f"'{path.name}' is empty -- not a dictionary."
    if path.suffix.lower() == ".dic" and text.count("%") < 2:
        return (
            f"'{path.name}' has no %...% category header, so it is a plain "
            "word list rather than a LIWC .dic. Add the header block "
            "(%, one 'number<TAB>name' per category, %) first."
        )
    if kind is not None and kind.deep_check is not None:
        return kind.deep_check(path)
    return ""

delete

delete(kind, name)

Remove an entry and the weights it carries. Permanent, so the UI confirms before calling this.

Source code in src\taters\helpers\library.py
966
967
968
969
970
971
972
973
def delete(kind: LibraryKind, name: str) -> None:
    """Remove an entry and the weights it carries. Permanent, so the UI
    confirms before calling this."""
    entry = _entry(kind, name)
    payloads = payload_of(entry)
    entry.unlink()
    for payload in payloads:
        _remove_payload(payload)

display_name

display_name(path)

How an entry is named in menus: the filename stem.

Source code in src\taters\helpers\library.py
574
575
576
def display_name(path: Path) -> str:
    """How an entry is named in menus: the filename stem."""
    return path.stem

entries

entries(kind)

Every entry of this kind, sorted by name. Only the kind's own formats count -- a stray .txt dropped into the folder by hand is ignored, not an error.

Source code in src\taters\helpers\library.py
560
561
562
563
564
565
566
567
568
569
570
571
def entries(kind: LibraryKind) -> List[Path]:
    """Every entry of this kind, sorted by name. Only the kind's own formats
    count -- a stray .txt dropped into the folder by hand is ignored, not an
    error."""
    folder = kind_dir(kind)
    carried = payload_dirs(folder)
    return sorted(
        (f for f in folder.iterdir()
         if f.is_file() and f.suffix.lower() in kind.suffixes
         and f not in carried),
        key=lambda f: f.name.lower(),
    )

expand

expand(kind, values)

Resolve a mixed list of files and folders to this kind's files.

A folder means "everything of this kind inside it, recursively" -- the reading the analyzers give a folder path, and what the wizard's default writes into a preset (the kind's whole library folder). Any screen seeded through this shows exactly what a run would use; seeding from the raw values made the picker intersect a folder path against entry file paths and open with everything unticked while the settings row said "all 10".

Source code in src\taters\helpers\library.py
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
def expand(kind: LibraryKind, values) -> List[Path]:
    """
    Resolve a mixed list of files and folders to this kind's files.

    A folder means "everything of this kind inside it, recursively" -- the
    reading the analyzers give a folder path, and what the wizard's default
    writes into a preset (the kind's whole library folder). Any screen seeded
    through this shows exactly what a run would use; seeding from the raw
    values made the picker intersect a folder path against entry file paths
    and open with everything unticked while the settings row said "all 10".
    """
    out: List[Path] = []
    for v in values:
        p = Path(v)
        if p.is_dir():
            out.extend(f for f in _files_under(p)
                       if f.suffix.lower() in kind.suffixes)
        else:
            out.append(p)
    return out

export_to

export_to(kind, name, dest_dir, *, replace=False)

Copy an entry out, keeping its filename. Returns the new path.

Raises:

Type Description
LibraryCollision

When the destination file already exists and replace is False -- the same contract as :func:import_into, for the same reason: a copy that silently overwrites is a delete nobody asked for.

Source code in src\taters\helpers\library.py
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
def export_to(kind: LibraryKind, name: str, dest_dir: Path,
              *, replace: bool = False) -> Path:
    """
    Copy an entry out, keeping its filename. Returns the new path.

    Raises
    ------
    LibraryCollision
        When the destination file already exists and ``replace`` is False --
        the same contract as :func:`import_into`, for the same reason: a copy
        that silently overwrites is a delete nobody asked for.
    """
    src = _entry(kind, name)
    dest_dir = Path(dest_dir)
    dest_dir.mkdir(parents=True, exist_ok=True)
    dest = dest_dir / src.name
    pairs = _payload_destinations(src, dest)
    if not replace:
        for taken in [dest] + [d for _s, d in pairs]:
            if taken.exists():
                raise LibraryCollision(dest)
    shutil.copy2(src, dest)
    for payload, target in pairs:
        _copy_payload(payload, target)
    return dest

find_payload

find_payload(kind, name, digests)

A payload in the library whose files match the digests a manifest records.

A model carried into another run travels as its manifest text only (see pipelines.run_pipeline._materialize_assets); the weights stay in the library. The manifest records payload_digests -- sha256 per file -- so the loader can find them here and prove they are the same bytes. The match is by content, not by name: a renamed library entry still counts. name is the payload's suffix-bearing name, used to skip payloads of the wrong shape quickly.

Source code in src\taters\helpers\library.py
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
def find_payload(kind: LibraryKind, name: str, digests: Mapping[str, str]) -> Optional[Path]:
    """
    A payload in the library whose files match the digests a manifest records.

    A model carried into another run travels as its manifest text only (see
    ``pipelines.run_pipeline._materialize_assets``); the weights stay in the
    library. The manifest records ``payload_digests`` -- sha256 per file --
    so the loader can find them here and prove they are the same bytes. The
    match is by content, not by name: a renamed library entry still counts.
    ``name`` is the payload's suffix-bearing name, used to skip payloads of
    the wrong shape quickly.
    """
    wanted = {str(k): str(v) for k, v in (digests or {}).items()}
    if not wanted:
        return None
    suffix = Path(name).suffix
    for manifest in entries(kind):
        for payload in payload_of(manifest):
            if suffix and payload.suffix != suffix:
                continue
            if not payload.exists():
                continue
            if _payload_matches(payload, wanted):
                return payload
    return None

import_into

import_into(kind, src, *, replace=False)

Copy a file into the library.

Raises:

Type Description
ValueError

For a format the kind does not accept -- with the formats it does, because "wrong extension" without the right ones is a dead end -- or for a file the kind's own parser cannot load (see :func:asset_problem), with the parser's reason.

LibraryCollision

When an entry with this name exists and replace is False. The UI turns this into a replace-or-keep question rather than deciding here.

Source code in src\taters\helpers\library.py
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
def import_into(kind: LibraryKind, src: Path, *, replace: bool = False) -> Path:
    """
    Copy a file into the library.

    Raises
    ------
    ValueError
        For a format the kind does not accept -- with the formats it does,
        because "wrong extension" without the right ones is a dead end -- or
        for a file the kind's own parser cannot load (see
        :func:`asset_problem`), with the parser's reason.
    LibraryCollision
        When an entry with this name exists and ``replace`` is False. The UI
        turns this into a replace-or-keep question rather than deciding here.
    """
    src = Path(src)
    if src.suffix.lower() not in kind.suffixes:
        raise ValueError(
            f"{src.name} is not a {kind.label.lower()} file "
            f"(accepted: {', '.join(kind.suffixes)})"
        )
    problem = asset_problem(src, kind)
    if problem:
        # we refuse right here, while the user still has the file in hand and
        # can fix it -- if we let it in, it blows up a run an hour later instead.
        raise ValueError(problem)
    missing = [p.name for p in payload_of(src) if not p.exists()]
    if missing:
        raise ValueError(
            f"{src.name} names weights that are not beside it "
            f"({', '.join(missing)}). Copy the model file and its weights "
            f"together, or re-run the step that made them.")
    dest = kind_dir(kind) / src.name
    pairs = _payload_destinations(src, dest)
    if not replace:
        for taken in [dest] + [d for _s, d in pairs]:
            if taken.exists():
                raise LibraryCollision(dest)
    shutil.copy2(src, dest)
    try:
        for payload, target in pairs:
            _copy_payload(payload, target)
        _rewrite_payload_names(dest, {s.name: d.name for s, d in pairs
                                      if s.name != d.name})
    except BaseException:
        # half an entry is worse than none: a manifest with no weights would get
        # listed, picked, and then refused at run time.
        dest.unlink(missing_ok=True)
        for _payload, target in pairs:
            _remove_payload(target)
        raise
    return dest

kind_by_id

kind_by_id(kind_id)

One kind, or a KeyError that names the valid ids (it is always a typo).

Source code in src\taters\helpers\library.py
379
380
381
382
383
384
385
386
def kind_by_id(kind_id: str) -> LibraryKind:
    """One kind, or a KeyError that names the valid ids (it is always a typo)."""
    try:
        return KINDS[kind_id]
    except KeyError:
        raise KeyError(
            f"unknown library kind {kind_id!r}. Known: {', '.join(sorted(KINDS))}"
        ) from None

kind_dir

kind_dir(kind)

This kind's folder, created and kept level with what the package ships.

Seeding used to happen only when the folder did not exist, which meant a release that added a dictionary reached new users and nobody else. So it now reconciles file by file against a ledger of what we have installed before (see :func:_seed), and an upgrade brings its new and corrected built-ins to a library that has been in use for years.

What it will never do is undo a decision: a built-in the user deleted stays deleted, and a copy they edited stays edited. An empty library is a valid state the UI explains, not one this quietly "repairs."

Source code in src\taters\helpers\library.py
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
def kind_dir(kind: LibraryKind) -> Path:
    """
    This kind's folder, created and kept level with what the package ships.

    Seeding used to happen only when the folder did not exist, which meant a
    release that added a dictionary reached new users and nobody else. So it
    now reconciles file by file against a ledger of what we have installed
    before (see :func:`_seed`), and an upgrade brings its new and corrected
    built-ins to a library that has been in use for years.

    What it will never do is undo a decision: a built-in the user deleted
    stays deleted, and a copy they edited stays edited. An empty library is a
    valid state the UI explains, not one this quietly "repairs."
    """
    target = library_home() / kind.id
    target.mkdir(parents=True, exist_ok=True)
    _seed(kind, target)
    return target

library_home

library_home()

Where the library lives: $TATERS_HOME/library, or ~/.taters/library.

The environment override exists so tests -- and anyone who wants their library on a different drive -- can move the whole thing without patching.

Source code in src\taters\helpers\library.py
389
390
391
392
393
394
395
396
397
def library_home() -> Path:
    """
    Where the library lives: ``$TATERS_HOME/library``, or ``~/.taters/library``.

    The environment override exists so tests -- and anyone who wants their
    library on a different drive -- can move the whole thing without patching.
    """
    base = os.environ.get("TATERS_HOME")
    return (Path(base) if base else Path.home() / ".taters") / "library"

model_files

model_files(folder)

The .json manifests under a folder, without the ones inside a payload.

What a step handed "a folder" resolves to, and what the finish screen scans a whole run folder with. Only .json files are walked -- a run over thousands of recordings holds tens of thousands of WAVs and CSVs, and listing every one of them to find a handful of manifests made the finish screen wait on a network drive. A payload declared by any manifest found is pruned, wherever in the tree it sits.

Source code in src\taters\helpers\library.py
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
def model_files(folder: Path) -> List[Path]:
    """
    The ``.json`` manifests under a folder, without the ones inside a payload.

    What a step handed "a folder" resolves to, and what the finish screen
    scans a whole run folder with. Only ``.json`` files are walked -- a run
    over thousands of recordings holds tens of thousands of WAVs and CSVs,
    and listing every one of them to find a handful of manifests made the
    finish screen wait on a network drive. A payload declared by any
    manifest found is pruned, wherever in the tree it sits.
    """
    folder = Path(folder)
    if not folder.is_dir():
        return []
    candidates = sorted(f for f in folder.rglob("*.json") if f.is_file())
    carried: Set[Path] = set()
    for f in candidates:
        carried.update(payload_of(f))
    return [f for f in candidates
            if f not in carried and not any(c in f.parents for c in carried)]

payload_dirs

payload_dirs(folder)

Every payload path declared by the manifests directly in folder.

Source code in src\taters\helpers\library.py
641
642
643
644
645
646
647
648
649
def payload_dirs(folder: Path) -> Set[Path]:
    """Every payload path declared by the manifests directly in ``folder``."""
    folder = Path(folder)
    if not folder.is_dir():
        return set()
    carried: Set[Path] = set()
    for manifest in folder.glob("*.json"):
        carried.update(payload_of(manifest))
    return carried

payload_of

payload_of(entry)

The sibling files and folders a .json entry declares as its payload.

Empty for anything that is not a JSON manifest with a payload list. Names are taken as siblings only: a name with a path separator, or a dot entry, is ignored rather than allowed to point outside the folder.

Source code in src\taters\helpers\library.py
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
def payload_of(entry: Path) -> List[Path]:
    """
    The sibling files and folders a ``.json`` entry declares as its payload.

    Empty for anything that is not a JSON manifest with a ``payload`` list.
    Names are taken as siblings only: a name with a path separator, or a
    dot entry, is ignored rather than allowed to point outside the folder.
    """
    entry = Path(entry)
    if entry.suffix.lower() != ".json" or not entry.is_file():
        return []
    try:
        doc = json.loads(entry.read_text(encoding="utf-8"))
    except (OSError, ValueError, UnicodeDecodeError):
        return []
    names = doc.get("payload") if isinstance(doc, dict) else None
    if not isinstance(names, list):
        return []
    out: List[Path] = []
    for name in names:
        text = str(name)
        if not text or "/" in text or "\\" in text or text in (".", ".."):
            continue
        out.append(entry.with_name(text))
    return out

payload_size

payload_size(entry)

Bytes of an entry and everything it carries.

Source code in src\taters\helpers\library.py
736
737
738
739
740
741
742
743
744
745
746
747
def payload_size(entry: Path) -> int:
    """Bytes of an entry and everything it carries."""
    total = 0
    for p in [Path(entry)] + payload_of(entry):
        try:
            if p.is_dir():
                total += sum(f.stat().st_size for f in p.rglob("*") if f.is_file())
            else:
                total += p.stat().st_size
        except OSError:
            continue
    return total

rename

rename(kind, old, new)

Rename an entry, keeping its suffix.

The suffix carries the format, which renaming must not be able to lie about -- so a new name arriving with an extension has it stripped rather than honored.

Source code in src\taters\helpers\library.py
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
def rename(kind: LibraryKind, old: str, new: str) -> Path:
    """
    Rename an entry, keeping its suffix.

    The suffix carries the *format*, which renaming must not be able to lie
    about -- so a new name arriving with an extension has it stripped rather
    than honored.
    """
    src = _entry(kind, old)
    if "/" in new or "\\" in new:
        # Path(new).stem quietly keeps only what follows the last separator:
        # "a/b" became "b", and a name ending "[/]" became "]". we found this
        # one by fuzzing; better to refuse than to mangle.
        raise ValueError(
            f"a name cannot contain '/' or '\\' (got {new!r})"
        )
    stem = Path(new).stem.strip()
    if not stem:
        raise ValueError("the new name is empty")
    dest = src.with_name(stem + src.suffix)
    pairs = _payload_destinations(src, dest)
    if dest != src:
        for taken in [dest] + [d for _s, d in pairs if d != _s]:
            if taken.exists():
                raise LibraryCollision(dest)
    src.rename(dest)
    for payload, target in pairs:
        if target != payload and payload.exists():
            payload.rename(target)
    _rewrite_payload_names(dest, {s.name: d.name for s, d in pairs
                                  if s.name != d.name})
    # a model's menus read the name *inside* its file, not the file's name
    # (see model_spec.describe_encoder): renaming the file alone left the
    # old name on every screen, which is how one once stayed
    # "roberta-base-adapted" through three renames
    _rewrite_doc_name(dest, stem)
    return dest

taters.helpers.model_spec

One description of a saved model, whatever kind it is.

Taters writes several kinds of reusable instrument -- a MEM topic model, a ridge regression, a classifier -- and they were each self-describing to their own scoring function and to nothing else. That was fine while each kind had its own menu entry, and stops being fine the moment a screen wants to offer models as a category: to list them it has to say what each one is, to score with one it has to know whether the thing needs text or features, and to write the results it has to know what columns will appear.

So each kind registers those three answers here, and every screen asks this module instead of sniffing JSON keys for itself. The alternative -- a per-kind branch at each call site -- is the shape where adding a fourth kind means finding all of them.

Adding a kind of model

Everything a screen or a scoring run needs to know about a model comes from this registry, so a new kind -- a fine-tuned transformer, a gradient-boosted tree, whatever comes next -- is added in one place:

  1. Give it a kind tag in the file it writes ("taters-<x>-model") and a format integer.
  2. Add a :class:ModelType to :data:MODEL_TYPES stating:

  3. needs -- "text" if it reads text and re-derives its own instrument (the MEM pattern), "features" if it reads feature columns someone else measured. That one word decides whether the settings-provenance gate applies to it, and nothing else has to know.

  4. read -- doc -> (outputs, columns, inputs), so listings can say what it is and what it will add.
  5. score -- "module:function" for applying it. Omit this and scoring refuses by name instead of guessing.
  6. a "payload" list in the model file -- the sibling weights the library moves with it (see helpers.library.payload_of)
  7. write_outputs -- if its output columns can be renamed at import.
  8. classes -- doc -> {outcome: [class, ...]} when it predicts categories, so a class can be relabeled per model ("0" -> "control") and every applier writes the label through :func:class_label.
  9. apply_settings -- the defaults that govern how it is applied to new data (batch size, weighting, whether to emit probabilities). They live in the model file's apply block, editable per model in Settings, and an applier reads them through :func:apply_defaults so a pipeline gets the model's own settings unless a call overrides them.

  10. Add a _load_model-style gate in the module that owns it, and register it in :func:taters.helpers.library._model_problem so the import screen is exactly as strict as run time.

Nothing else needs editing. In particular the scoring entry point, the library folder, the naming flow, the wizard row and the provenance gate are all written against this registry rather than against the three kinds that happen to exist today.

Naming

A model file is named by whoever exported it, which is nearly always the step's own output name (ridge__all__age.json), and that says nothing about what it predicts. So a model carries a name the researcher chose, and output names for the columns it will write: a ridge fitted to predict age on a blog corpus should be able to land in a new dataset as pred_age_blogs, not as pred_age colliding with the age already there. Both are stored in the model file, so they travel with it.

ApplySetting dataclass

ApplySetting(
    default, help, kind="str", choices=None, validate=None
)

One default that governs how a kind of model is applied to new data.

Stored per model in the file's apply block and read back through :func:apply_defaults, so a model carries its own settings wherever it goes -- a word-vector model that should weight by types, a predictor that fits a small card at batch 8 -- rather than every pipeline having to know. kind says how a typed-in value is read: "int", "float", "bool", "str", or "text" for free text that is not one of a fixed set. choices restricts a str to a list.

coerce

coerce(value, name)

The value as this setting stores it, or a refusal naming both.

Source code in src\taters\helpers\model_spec.py
230
231
232
233
234
235
def coerce(self, value, name: str):
    """The value as this setting stores it, or a refusal naming both."""
    out = self._coerce(value, name)
    if self.validate is not None:
        self.validate(out)
    return out

FeaturePlan dataclass

FeaturePlan(
    model,
    slug,
    tables=(),
    problems=(),
    path="",
    controls=(),
)

Everything a run needs in order to score with one model, as plain data.

Deliberately free of paths, recipes and pipeline templates: it is built from the model file alone, so the composer stays pure and a test can hand a synthetic plan straight in.

problems is the honest half. A plan with problems cannot be replayed, and each entry is a sentence a researcher can act on -- which is better than a partial replay, because a partial replay is a wrong answer.

ModelInfo dataclass

ModelInfo(
    path,
    type_id,
    type_label,
    name,
    outputs,
    columns,
    inputs,
    needs,
    needs_tables=(),
    provenance=dict(),
    library_kind=LIBRARY_KIND,
    bulk_outputs=False,
    controls=(),
    zero_when_absent=(),
    classes=dict(),
    class_names=dict(),
    apply=dict(),
    modality="text",
)

What one saved model is, in the terms a screen needs.

Attributes:

Name Type Description
path Path

The file itself.

type_id str

Short kind key: "ridge", "classifier", "mem".

type_label str

What to show a person: "ridge", "MEM topic model". Menus put it in brackets after the name, so a library listing reads age_blogs [ridge] and the two are told apart at a glance.

name str

The researcher's name for the model, defaulting to the file stem.

outputs tuple of str

The output labels, after any renaming -- one per predicted outcome, or one per theme.

columns tuple of str

The column names those outputs will actually produce, which is not the same list: a classifier writes a predicted class and a probability per class for one outcome.

inputs tuple of str

Feature columns the model needs by name. Empty when the model works from text.

needs str

"features" or "text" -- what has to exist upstream before this model can score anything.

needs_tables tuple of str

Which feature tables, named by the step that writes them. A model records its predictors by name, which is enough to score a table that already has them and no help at all in getting there: someone who fitted on cohesion features and came back a week later had 165 column names and nothing saying a cohesion step was needed.

library_kind str

The :data:~taters.helpers.library.KINDS id this model imports as.

bulk_outputs bool

True when the outputs are a numbered family (MEM themes) rather than a handful of named quantities, so renaming them means choosing one prefix rather than editing a hundred labels.

n_outputs int

How many outputs there are, for a listing that does not want to print a hundred theme names.

display

display()

age_blogs [ridge] -- the one-line form every menu uses.

Source code in src\taters\helpers\model_spec.py
203
204
205
def display(self) -> str:
    """``age_blogs [ridge]`` -- the one-line form every menu uses."""
    return f"{self.name} [{self.type_label}]"

ModelType dataclass

ModelType(
    id,
    label,
    kind_tag,
    needs,
    read,
    score=None,
    write_outputs=None,
    bulk_outputs=False,
    classes=None,
    apply_settings=dict(),
    modality="text",
)

One kind of model, and how to read and rewrite its names.

TablePlan dataclass

TablePlan(
    stem,
    target,
    instrument,
    assets,
    digest,
    grain,
    replay=None,
)

One feature table a model needs, and exactly how to measure it.

Attributes:

Name Type Description
stem str

The table's name at fit time. Load-bearing: the stem is what fixed the model's predictor names, so a replayed table has to be written under it or the names stop matching.

target str

"module:function" of the analyzer, as recorded. Byte-identical to the producing recipe's target, which is how a composer joins the two without knowing anything about models.

instrument dict

The measuring settings, as literal values. Complete, because they were recorded after defaults were applied -- which is the whole reason a replay is constructible rather than a guess.

assets dict

{parameter: [{name, sha256, text}]} for the word lists it used, carried inside the model so the replay works on a machine that never had them.

digest str

The instrument digest. Belongs in the private output directory: two models needing identical settings then share one extraction for free, and a re-fit under the same model name cannot short-circuit onto the previous private table.

grain dict

What one row was at fit time. Informational only -- never compared.

replay dict or None

{"target": "module:function", "assets": {parameter: [...]}} when the table cannot be measured the same way twice and must instead be produced by applying what the fit-time step fitted -- a topic model's saved themes. The assets carry the fitted file's text, the same way assets carries word lists. None for a table whose settings alone reproduce it.

UnknownModel

UnknownModel(message, kind=None)

Bases: Exception

The file is not a saved Taters model, or is a kind this build cannot describe. Carries the kind tag it did have, when it had one.

Source code in src\taters\helpers\model_spec.py
102
103
104
def __init__(self, message: str, kind: Optional[str] = None):
    self.kind = kind
    super().__init__(message)

apply_defaults

apply_defaults(doc)

How a model wants to be applied: registry defaults under its own.

Only the settings its kind registers are read, each coerced to its type; a key the file carries from a newer Taters is ignored rather than passed on. An applier takes apply_defaults(doc)[key] wherever its caller left the argument as None, so the model's own settings win over the function's signature but never over an explicit call.

Source code in src\taters\helpers\model_spec.py
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
def apply_defaults(doc: dict) -> Dict[str, object]:
    """
    How a model wants to be applied: registry defaults under its own.

    Only the settings its kind registers are read, each coerced to its
    type; a key the file carries from a newer Taters is ignored rather than
    passed on. An applier takes ``apply_defaults(doc)[key]`` wherever its
    caller left the argument as None, so the model's own settings win over
    the function's signature but never over an explicit call.
    """
    spec = MODEL_TYPES.get(str(doc.get("kind")))
    if spec is None or not spec.apply_settings:
        return {}
    stored = doc.get("apply") or {}
    out: Dict[str, object] = {}
    for key, setting in spec.apply_settings.items():
        value = setting.default
        if key in stored:
            try:
                value = setting.coerce(stored[key], key)
            except ValueError:
                value = setting.default    # a damaged value shouldn't block scoring
        out[key] = value
    return out

class_label

class_label(doc, outcome, cls)

The label written for one predicted class: its relabel, or itself.

A model fitted on a condition column coded 0/1 predicts "0" and "1" -- correct, and unreadable in a results table a month later. The relabel lives in the model file (class_names), keyed by the class as fitted, and every applier writes predictions and the p_<outcome>_<class> columns through this one function, so a model relabeled in Settings is relabeled however it is invoked.

Source code in src\taters\helpers\model_spec.py
316
317
318
319
320
321
322
323
324
325
326
327
328
329
def class_label(doc: dict, outcome: str, cls) -> str:
    """
    The label written for one predicted class: its relabel, or itself.

    A model fitted on a ``condition`` column coded 0/1 predicts ``"0"`` and
    ``"1"`` -- correct, and unreadable in a results table a month later.
    The relabel lives in the model file (``class_names``), keyed by the
    class as fitted, and every applier writes predictions and the
    ``p_<outcome>_<class>`` columns through this one function, so a model
    relabeled in Settings is relabeled however it is invoked.
    """
    names = (doc.get("class_names") or {}).get(outcome) or {}
    label = names.get(str(cls))
    return str(label) if label else str(cls)

describe

describe(model_json)

Describe one saved model, or say why it cannot be described.

Deliberately cheap and structural: it reads the file's own declaration of what it is and does not vet the matrices. Vetting is the scoring loader's job (and the library import gate delegates to it), because a listing that had to fully validate every model would be slow and would hide a damaged model behind a blank row instead of naming it.

Source code in src\taters\helpers\model_spec.py
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
def describe(model_json: PathLike) -> ModelInfo:
    """
    Describe one saved model, or say why it cannot be described.

    Deliberately cheap and structural: it reads the file's own declaration
    of what it is and does not vet the matrices. Vetting is the scoring
    loader's job (and the library import gate delegates to it), because a
    listing that had to fully validate every model would be slow and would
    hide a damaged model behind a blank row instead of naming it.
    """
    path = Path(model_json)
    doc = _read_doc(path)
    tag = doc.get("kind")
    spec = MODEL_TYPES.get(tag) if isinstance(tag, str) else None
    if spec is None:
        raise UnknownModel(
            f"{path.name} is "
            + (f"a {tag!r} file, which is not a model this build can score"
               if tag else "not a Taters model file at all")
            + ". Models are written by the topic-model, prediction and "
              "classification steps.",
            kind=tag if isinstance(tag, str) else None)
    outputs, columns, inputs = spec.read(doc)
    fitted = dict(spec.classes(doc)) if spec.classes is not None else {}
    relabels = doc.get("class_names") or {}
    return ModelInfo(
        classes={o: tuple(class_label(doc, o, c) for c in cs)
                 for o, cs in fitted.items()},
        class_names={o: {str(k): str(v) for k, v in (m or {}).items()}
                     for o, m in relabels.items() if o in fitted},
        apply=apply_defaults(doc),
        path=path, type_id=spec.id, type_label=spec.label,
        name=str(doc.get("name") or path.stem),
        outputs=tuple(outputs), columns=tuple(columns), inputs=tuple(inputs),
        needs=spec.needs, bulk_outputs=spec.bulk_outputs, modality=spec.modality,
        provenance=((doc.get("needs") or {}).get("feature_provenance") or {}),
        controls=tuple(doc.get("controls") or ()),
        zero_when_absent=tuple(str(c) for c in
                               (doc.get("zero_when_absent") or ())),
        needs_tables=tuple(
            str(s) for s in ((doc.get("needs") or {}).get("feature_tables")
                             or ())))

describe_all

describe_all(paths)

Describe every model that can be described, skipping the rest silently.

Used by listings, where one unreadable file in a library folder must not take the whole menu down -- the import gate already refused anything broken, so a file that fails here arrived some other way.

Source code in src\taters\helpers\model_spec.py
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
def describe_all(paths: Sequence[PathLike]) -> List[ModelInfo]:
    """
    Describe every model that can be described, skipping the rest silently.

    Used by listings, where one unreadable file in a library folder must not
    take the whole menu down -- the import gate already refused anything
    broken, so a file that fails here arrived some other way.
    """
    out = []
    for path in paths:
        try:
            out.append(describe(path))
        except (UnknownModel, FileNotFoundError, OSError):
            continue
    return out

describe_encoder

describe_encoder(model_json)

One encoder's row: its name and where it came from, then what training did.

Source code in src\taters\helpers\model_spec.py
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
def describe_encoder(model_json: PathLike) -> Tuple[str, str]:
    """One encoder's row: its name and where it came from, then what training did."""
    path = Path(model_json)
    doc = _read_doc(path)
    name = str(doc.get("name") or path.stem)
    # an encoder trained from scratch has no base model, and "[?]" would read
    # as a broken file rather than as the deliberate absence it is
    if doc.get("trained_from") == "scratch":
        base = "trained from scratch"
    else:
        base = str(doc.get("base_model") or "?")
    bits = []
    layers = doc.get("num_hidden_layers")
    if layers:
        bits.append(f"{layers} layers")
    ev = doc.get("evaluation") or {}
    before, after = ev.get("perplexity_before"), ev.get("perplexity_after")
    if before is not None and after is not None:
        bits.append(f"perplexity {float(before):.1f} → {float(after):.1f}")
    elif ev.get("perplexity_final") is not None:
        bits.append(f"perplexity {float(ev['perplexity_final']):.1f}")
    return f"{name} [{base}]", " · ".join(bits)

edit_model

edit_model(
    model_json,
    *,
    name=None,
    outputs=None,
    prefix=None,
    class_names=None,
    apply=None
)

Change what a model is called, what it writes, and how it is applied.

Parameters:

Name Type Description Default
model_json PathLike

The model file, rewritten in place (atomically).

required
name Optional[str]

The researcher's name for the model. Shown in every menu.

None
outputs Optional[Sequence[str]]

One new label per output, in the order :func:describe lists them.

None
prefix Optional[str]

For a model whose outputs are a numbered family (MEM themes), the stem to number from: "fb_topics" gives fb_topics_1 upward. Cannot be combined with outputs.

None
class_names Optional[Mapping[str, Mapping[str, str]]]

{outcome: {class as fitted: label to write}} for a model that predicts categories. Merged with what the file already has, one outcome at a time: a class left out keeps its current label, and giving a class its own name (or a blank) removes its relabel. Two classes of one outcome cannot end up with the same label.

None
apply Optional[Mapping[str, object]]

Values for the settings its kind registers (:attr:ModelType. apply_settings), coerced and checked; an unknown key is refused naming the ones there are. Merged with the file's apply block.

None

Returns:

Type Description
ModelInfo

The model as it now reads.

Notes

Every edit rewrites the file rather than a sidecar, so a model that is copied, zipped or emailed keeps the names and settings it was given -- a sidecar would be left behind by every one of those, and the columns would quietly revert.

Source code in src\taters\helpers\model_spec.py
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
def edit_model(model_json: PathLike, *, name: Optional[str] = None,
               outputs: Optional[Sequence[str]] = None,
               prefix: Optional[str] = None,
               class_names: Optional[Mapping[str, Mapping[str, str]]] = None,
               apply: Optional[Mapping[str, object]] = None) -> ModelInfo:
    """
    Change what a model is called, what it writes, and how it is applied.

    Parameters
    ----------
    model_json
        The model file, rewritten in place (atomically).
    name
        The researcher's name for the model. Shown in every menu.
    outputs
        One new label per output, in the order :func:`describe` lists them.
    prefix
        For a model whose outputs are a numbered family (MEM themes), the
        stem to number from: ``"fb_topics"`` gives ``fb_topics_1`` upward.
        Cannot be combined with ``outputs``.
    class_names
        ``{outcome: {class as fitted: label to write}}`` for a model that
        predicts categories. Merged with what the file already has, one
        outcome at a time: a class left out keeps its current label, and
        giving a class its own name (or a blank) removes its relabel. Two
        classes of one outcome cannot end up with the same label.
    apply
        Values for the settings its kind registers (:attr:`ModelType.
        apply_settings`), coerced and checked; an unknown key is refused
        naming the ones there are. Merged with the file's ``apply`` block.

    Returns
    -------
    ModelInfo
        The model as it now reads.

    Notes
    -----
    Every edit rewrites the file rather than a sidecar, so a model that is
    copied, zipped or emailed keeps the names and settings it was given --
    a sidecar would be left behind by every one of those, and the columns
    would quietly revert.
    """
    path = Path(model_json)
    doc = _read_doc(path)
    info = describe(path)
    spec = BY_ID[info.type_id]

    if class_names is not None:
        fitted = dict(spec.classes(doc)) if spec.classes is not None else {}
        if not fitted:
            raise ValueError(
                f"a {info.type_label} model predicts no categories, so it "
                f"has no classes to relabel")
        current = {o: dict(m or {}) for o, m in
                   (doc.get("class_names") or {}).items()}
        for outcome, mapping in class_names.items():
            if outcome not in fitted:
                raise ValueError(
                    f"this model has no categorical outcome {outcome!r} "
                    f"(it has {', '.join(fitted) or 'none'})")
            table = current.setdefault(outcome, {})
            for cls, label in (mapping or {}).items():
                cls = str(cls)
                if cls not in [str(c) for c in fitted[outcome]]:
                    raise ValueError(
                        f"outcome {outcome!r} has no class {cls!r} (its "
                        f"classes are {', '.join(map(str, fitted[outcome]))})")
                text = str(label if label is not None else "").strip()
                if not text or text == cls:
                    table.pop(cls, None)
                else:
                    table[cls] = _valid_label(text, what="class label")
            written = [table.get(str(c), str(c)) for c in fitted[outcome]]
            clashes = sorted({w for w in written if written.count(w) > 1})
            if clashes:
                raise ValueError(
                    f"two classes of {outcome!r} cannot share the label "
                    f"{clashes[0]!r} -- the results could not tell them apart")
        doc["class_names"] = {o: m for o, m in current.items() if m}
        if not doc["class_names"]:
            doc.pop("class_names", None)

    if apply is not None:
        if not spec.apply_settings:
            raise ValueError(
                f"a {info.type_label} model has no settings for how it is "
                f"applied")
        block = dict(doc.get("apply") or {})
        for key, value in apply.items():
            setting = spec.apply_settings.get(str(key))
            if setting is None:
                raise ValueError(
                    f"{key!r} is not a setting of a {info.type_label} model "
                    f"(they are: {', '.join(spec.apply_settings)})")
            block[str(key)] = setting.coerce(value, str(key))
        doc["apply"] = block

    if outputs is not None and prefix is not None:
        raise ValueError(
            "give either one label per output or a single prefix, not both")
    if prefix is not None:
        if not info.bulk_outputs:
            raise ValueError(
                f"a {info.type_label} model has {info.n_outputs} named "
                f"output(s), so name them individually rather than by "
                f"prefix")
        outputs = [f"{prefix}_{i + 1}" for i in range(info.n_outputs)]
    if outputs is not None:
        labels = [_valid_label(label) for label in outputs]
        duplicates = sorted({label for label in labels
                             if labels.count(label) > 1})
        if duplicates:
            raise ValueError(
                f"two outputs cannot share a name ({', '.join(duplicates)}) "
                f"-- one column would overwrite the other")
        if spec.write_outputs is None:
            raise ValueError(
                f"a {info.type_label} model's output names cannot be changed")
        spec.write_outputs(doc, labels)
    if name is not None:
        doc["name"] = _valid_label(name, what="model name")

    with atomic_write(path, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)
    return replace(describe(path), path=path)

encoder_problem

encoder_problem(model_json)

Why this file is not a usable text encoder, or "" when it is.

Structural and torch-free: the manifest's kind and format, and a payload folder beside it holding a config, weights and a tokenizer. Loading the weights is the extractor's job, minutes later, on the device it chose.

Source code in src\taters\helpers\model_spec.py
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
def encoder_problem(model_json: PathLike) -> str:
    """
    Why this file is not a usable text encoder, or "" when it is.

    Structural and torch-free: the manifest's kind and format, and a payload
    folder beside it holding a config, weights and a tokenizer. Loading the
    weights is the extractor's job, minutes later, on the device it chose.
    """
    path = Path(model_json)
    try:
        doc = _read_doc(path)
    except UnknownModel as e:
        return str(e)
    except FileNotFoundError as e:
        return str(e)
    if doc.get("kind") != ENCODER_KIND:
        return (f"{path.name} is not a text encoder "
                f"(a {doc.get('kind')!r} file).")
    try:
        fmt = int(doc.get("format", 0))
    except (TypeError, ValueError):
        return f"{path.name} has no readable format number."
    if fmt > ENCODER_FORMAT:
        return (f"{path.name} was written by a newer Taters (format {fmt}); "
                f"this build reads format {ENCODER_FORMAT}. Update Taters.")
    from .library import payload_of

    declared = payload_of(path)
    if not declared:
        return (f"{path.name} names no weights folder beside it. The "
                f"encoder's weights folder has to travel with the file.")
    folder = declared[0]
    if not folder.is_dir():
        return (f"{path.name}'s weights folder {folder.name} is not beside "
                f"it. Copy the model file and its weights together.")
    names = {f.name for f in folder.iterdir()}
    for needed in _ENCODER_FILES:
        if needed not in names:
            return f"{folder.name} has no {needed}; the encoder is incomplete."
    if not names & set(_ENCODER_WEIGHTS):
        return f"{folder.name} has no model weights; the encoder is incomplete."
    if not names & set(_ENCODER_TOKENIZER):
        return f"{folder.name} has no tokenizer; the encoder is incomplete."
    return ""

feature_plan

feature_plan(info)

Read a model's own account of the features it was fitted on.

Returns a :class:FeaturePlan. A model that reads text needs no features and gets an empty plan with no problems -- it re-derives its own instrument, so there is nothing for a run to arrange.

Source code in src\taters\helpers\model_spec.py
 974
 975
 976
 977
 978
 979
 980
 981
 982
 983
 984
 985
 986
 987
 988
 989
 990
 991
 992
 993
 994
 995
 996
 997
 998
 999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
def feature_plan(info: ModelInfo) -> FeaturePlan:
    """
    Read a model's own account of the features it was fitted on.

    Returns a :class:`FeaturePlan`. A model that reads text needs no features
    and gets an empty plan with no problems -- it re-derives its own
    instrument, so there is nothing for a run to arrange.
    """
    import json

    name_slug = slug(info.name)
    controls = tuple(sorted({str(e.get("column")) for e in info.controls
                             if e.get("column")}))
    if info.needs != "features":
        return FeaturePlan(model=info.display(), slug=name_slug,
                           path=str(info.path))

    recorded = dict(info.provenance or {})
    if not recorded:
        return FeaturePlan(
            model=info.display(), slug=name_slug, path=str(info.path),
            controls=controls,
            problems=(f"{info.display()} does not record how its features "
                      f"were measured, so they cannot be reproduced.",))

    try:
        with Path(info.path).open("r", encoding="utf-8") as fh:
            carried = json.load(fh).get("assets") or {}
    except (OSError, json.JSONDecodeError, UnicodeDecodeError):
        carried = {}

    tables, problems = [], []
    for stem, rec in sorted(recorded.items()):
        if rec.get("state") != "recorded":
            problems.append(
                f"{info.display()} was fitted on the {stem!r} feature table "
                f"and has no record of how it was measured, so that table "
                f"cannot be reproduced.")
            continue
        assets, missing = _carried(rec.get("assets"), carried)
        replay = None
        if rec.get("replay"):
            fitted, gone = _carried((rec["replay"] or {}).get("assets"),
                                    carried)
            missing += gone
            replay = {"target": str((rec["replay"] or {}).get("call") or ""),
                      "assets": fitted}
        if missing:
            problems.append(
                f"{info.display()} used word list(s) it does not carry "
                f"({', '.join(sorted(set(missing)))}), so the {stem!r} table "
                f"cannot be reproduced. Fit the model again with a build of "
                f"Taters that embeds them.")
            continue
        tables.append(TablePlan(
            stem=str(stem), target=str(rec.get("call") or ""),
            instrument=dict(rec.get("instrument") or {}),
            assets=assets,
            digest=str((rec.get("digests") or {}).get("instrument") or ""),
            grain=dict(rec.get("grain") or {}), replay=replay))
    return FeaturePlan(model=info.display(), slug=name_slug,
                       tables=tuple(tables), problems=tuple(problems),
                       path=str(info.path), controls=controls)

library_kind_for

library_kind_for(model_json)

Which library kind a model file belongs in: encoders for an adapted encoder, models for anything that scores.

Source code in src\taters\helpers\model_spec.py
1185
1186
1187
1188
1189
1190
1191
1192
def library_kind_for(model_json: PathLike) -> str:
    """Which library kind a model file belongs in: ``encoders`` for an
    adapted encoder, ``models`` for anything that scores."""
    try:
        doc = _read_doc(Path(model_json))
    except (UnknownModel, FileNotFoundError, OSError):
        return "models"
    return "encoders" if doc.get("kind") == ENCODER_KIND else "models"

models_produced

models_produced(folder)

The model files a run left behind, each with a label for a menu.

Scoring models of every registered kind and adapted encoders alike, found by their manifests (never inside a payload folder), so the finish screen and the training task can offer to add them to the library.

Source code in src\taters\helpers\model_spec.py
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
def models_produced(folder: PathLike) -> List[Tuple[Path, str]]:
    """
    The model files a run left behind, each with a label for a menu.

    Scoring models of every registered kind and adapted encoders alike,
    found by their manifests (never inside a payload folder), so the finish
    screen and the training task can offer to add them to the library.
    """
    from .library import model_files

    out: List[Tuple[Path, str]] = []
    for path in model_files(Path(folder)):
        try:
            doc = _read_doc(path)
        except (UnknownModel, FileNotFoundError, OSError):
            continue
        if doc.get("kind") == ENCODER_KIND:
            label, _note = describe_encoder(path)
            out.append((path, f"{label} (text encoder)"))
            continue
        try:
            info = describe(path)
        except (UnknownModel, FileNotFoundError, OSError):
            continue
        out.append((path, info.display()))
    return out

one_model_path

one_model_path(model_json)

Exactly one model file, or an actionable refusal.

The library hands steps a folder (or a list) meaning "everything of this kind", which is right for dictionaries and wrong for a model: scoring with an unspecified one of three is not a thing anyone means. A single file passes through, a folder or a list resolves to its .json files, and anything other than exactly one refuses with the fix named.

Owned here because it is about model files, not about any one kind of model: ridge and the topic model each carried a copy, with different refusal wording and one of them letting a folder through unresolved.

Source code in src\taters\helpers\model_spec.py
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
def one_model_path(model_json) -> Path:
    """
    Exactly one model file, or an actionable refusal.

    The library hands steps a folder (or a list) meaning "everything of this
    kind", which is right for dictionaries and wrong for a model: scoring
    with an unspecified one of three is not a thing anyone means. A single
    file passes through, a folder or a list resolves to its ``.json`` files,
    and anything other than exactly one refuses with the fix named.

    Owned here because it is about model files, not about any one kind of
    model: ridge and the topic model each carried a copy, with different
    refusal wording and one of them letting a folder through unresolved.
    """
    from .library import model_files

    values = list(model_json) if isinstance(model_json, (list, tuple)) \
        else [model_json]
    found: List[Path] = []
    for value in values:
        path = Path(value)
        if path.is_dir():
            # not `rglob`: a checkpoint folder inside a model's payload holds
            # a config.json and a tokenizer.json that aren't model files.
            found += model_files(path)
        else:
            found.append(path)
    if len(found) == 1:
        return found[0]
    if not found:
        raise ValueError(
            f"no model file found at {model_json!r}. Import one into your "
            f"library, or point the step at one.")
    names = ", ".join(p.name for p in found[:6])
    raise ValueError(
        f"{len(found)} model files found ({names}) and a step can only use "
        f"one. Pick it in the step's options.")

output_label

output_label(doc, outcome)

The column-name stem for one outcome: its rename, or the outcome itself.

Read by the scoring functions rather than by the wizard, so a renamed model produces renamed columns however it is invoked -- through the app, through the API, or from the command line.

Source code in src\taters\helpers\model_spec.py
303
304
305
306
307
308
309
310
311
312
313
def output_label(doc: dict, outcome: str) -> str:
    """
    The column-name stem for one outcome: its rename, or the outcome itself.

    Read by the scoring functions rather than by the wizard, so a renamed
    model produces renamed columns however it is invoked -- through the app,
    through the API, or from the command line.
    """
    names = doc.get("output_names") or {}
    label = names.get(outcome)
    return str(label) if label else str(outcome)

rename_model

rename_model(
    model_json, *, name=None, outputs=None, prefix=None
)

Name a model and its output columns -- :func:edit_model without the class labels and apply settings. Kept for the callers that only ever name things.

Source code in src\taters\helpers\model_spec.py
859
860
861
862
863
864
865
def rename_model(model_json: PathLike, *, name: Optional[str] = None,
                 outputs: Optional[Sequence[str]] = None,
                 prefix: Optional[str] = None) -> ModelInfo:
    """Name a model and its output columns -- :func:`edit_model` without
    the class labels and apply settings. Kept for the callers that only
    ever name things."""
    return edit_model(model_json, name=name, outputs=outputs, prefix=prefix)

scorer

scorer(type_id)

The function that scores a new table with this kind of model.

Resolved from the registry rather than chosen at the call site, so that adding a kind of model is adding a registry entry -- and forgetting the entry is this refusal, rather than the model being quietly scored by whichever applier happened to be the else branch.

Source code in src\taters\helpers\model_spec.py
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
def scorer(type_id: str):
    """
    The function that scores a new table with this kind of model.

    Resolved from the registry rather than chosen at the call site, so that
    adding a kind of model is adding a registry entry -- and forgetting the
    entry is this refusal, rather than the model being quietly scored by
    whichever applier happened to be the `else` branch.
    """
    import importlib

    spec = BY_ID.get(type_id)
    if spec is None or not spec.score:
        raise ValueError(
            f"no scoring function is registered for a {type_id!r} model. Add "
            f"`score=` to its entry in MODEL_TYPES -- see 'Adding a kind of "
            f"model' in this module's docstring.")
    module, _, name = spec.score.partition(":")
    return getattr(importlib.import_module(module), name)

slug

slug(name, fallback='model')

Filesystem- and template-safe: artifact references split on . and :, and a save_as built from this must not contain either.

One spelling for every model-shaped file name. Three private copies had grown -- here, in ridge (fallback "set") and in score_model -- and two of them kept the dot the third refused, so the same set name could produce two different file names depending on which module wrote it.

Source code in src\taters\helpers\model_spec.py
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
def slug(name: str, fallback: str = "model") -> str:
    """
    Filesystem- and template-safe: artifact references split on ``.`` and
    ``:``, and a ``save_as`` built from this must not contain either.

    One spelling for every model-shaped file name. Three private copies had
    grown -- here, in ridge (fallback "set") and in score_model -- and two of
    them kept the dot the third refused, so the same set name could produce
    two different file names depending on which module wrote it.
    """
    import re

    return re.sub(r"[^0-9A-Za-z_-]+", "-", str(name)).strip("-") or fallback

taters.helpers.settings

Persistent user settings, and the one that matters today: where downloaded models are kept.

Taters has a home folder ($TATERS_HOME or ~/.taters) for the library; this module puts a small settings.json beside it for choices that should outlive a session and are not about any one pipeline. The first such choice is the model cache: every transformer, sentence-transformers and Whisper model is downloaded by the Hugging Face hub library into its cache, which lives under the user's home by default. On a shared server that is the wrong place -- one copy per user of a 500 MB encoder, on a home partition that is small on purpose -- and the fix has been an environment variable that has to be set in every shell. Here it is a setting, chosen once in Settings and applied every time Taters starts, before any of those libraries is imported.

Resolution order for the model cache, most explicit first:

  1. TATERS_MODEL_CACHE in the environment (an administrator's or a test's word, and the one thing that beats a saved setting);
  2. model_cache in settings.json (the user's choice in Settings);
  3. HF_HUB_CACHE or HF_HOME from the environment (the hub library's own conventions, honored as they always were);
  4. the hub library's default, ~/.cache/huggingface/hub.

Whichever wins is exported as HF_HUB_CACHE at import (see :func:apply_model_cache), so the hub library, transformers, sentence-transformers and faster-whisper all download to and read from the same place -- and it is passed explicitly to the transformer steps too, in case those libraries were imported before Taters was.

apply_model_cache

apply_model_cache()

Export the chosen cache as HF_HUB_CACHE so every downloading library agrees with it. Called when Taters is imported. Returns the path when a Taters-level choice (environment or setting) was applied, else None -- the hub library's own environment is left exactly as found.

Source code in src\taters\helpers\settings.py
164
165
166
167
168
169
170
171
172
173
174
175
176
177
def apply_model_cache() -> Optional[Path]:
    """
    Export the chosen cache as ``HF_HUB_CACHE`` so every downloading library
    agrees with it. Called when Taters is imported. Returns the path when a
    Taters-level choice (environment or setting) was applied, else None --
    the hub library's own environment is left exactly as found.
    """
    path, source = model_cache_source()
    if source not in ("environment", "setting"):
        return None
    os.environ["HF_HUB_CACHE"] = str(path)
    # older transformers releases read this name; it's harmless on newer ones.
    os.environ.setdefault("TRANSFORMERS_CACHE", str(path))
    return path

clear_setting

clear_setting(key)

Forget one setting; nothing happens when it was not set.

Source code in src\taters\helpers\settings.py
116
117
118
119
120
121
122
123
124
125
126
127
def clear_setting(key: str) -> None:
    """Forget one setting; nothing happens when it was not set."""
    from .atomic import atomic_write

    doc = load_settings()
    if str(key) not in doc:
        return
    del doc[str(key)]
    path = settings_path()
    path.parent.mkdir(parents=True, exist_ok=True)
    with atomic_write(path, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)

describe_model_cache

describe_model_cache()

One line for the setup check: the folder, how it was chosen, and how much is in it.

Source code in src\taters\helpers\settings.py
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
def describe_model_cache() -> str:
    """One line for the setup check: the folder, how it was chosen, and
    how much is in it."""
    path, source = model_cache_source()
    how = {"environment": f"from {MODEL_CACHE_ENV}",
           "setting": "chosen in Settings",
           "hub environment": "from HF_HUB_CACHE / HF_HOME",
           "default": "the default"}[source]
    if not path.is_dir():
        return f"{path}  ·  {how}  ·  nothing downloaded yet"
    models = [p for p in path.glob("models--*") if p.is_dir()]
    # a snapshot file is a link to its blob on Linux and macOS: counting both
    # said 30 GB for a 16 GB cache. a Windows cache without links holds plain
    # files in the snapshots and nothing in blobs, so it is counted once too
    size = sum(f.stat().st_size for f in path.rglob("*") if f.is_file() and not f.is_symlink())
    from .library import _human_size

    return f"{path}  ·  {how}  ·  {len(models)} model(s), {_human_size(size)}"

inspect_row_limit

inspect_row_limit()

How many rows to read when looking a spreadsheet over. 0 means all.

All of them by default, because the questions the wizard builds from a spreadsheet are only as true as what it read: on a real file, ten columns looked constant within a group across the first two hundred rows and were not across the other seven hundred, so they were offered as controls that would have come out empty. Reading everything also says at the moment the file is chosen whether its rows match its header, which is much cheaper to learn then than after an hour of extraction.

Lowered by somebody whose files are big enough that a full pass is worth skipping, in Settings. The environment variable wins, for scripts and tests.

Source code in src\taters\helpers\settings.py
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
def inspect_row_limit() -> int:
    """
    How many rows to read when looking a spreadsheet over. ``0`` means all.

    All of them by default, because the questions the wizard builds from a
    spreadsheet are only as true as what it read: on a real file, ten
    columns looked constant within a group across the first two hundred rows
    and were not across the other seven hundred, so they were offered as
    controls that would have come out empty. Reading everything also says at
    the moment the file is chosen whether its rows match its header, which
    is much cheaper to learn then than after an hour of extraction.

    Lowered by somebody whose files are big enough that a full pass is worth
    skipping, in Settings. The environment variable wins, for scripts and
    tests.
    """
    raw = os.environ.get(INSPECT_ROWS_ENV)
    if raw is None:
        raw = load_settings().get(INSPECT_ROWS_KEY)
    if raw is None or str(raw).strip() == "":
        return 0
    try:
        return max(0, int(float(str(raw).strip())))
    except ValueError:
        # a hand-edited settings file with "lots" in it is not a reason to
        # refuse to start, and reading everything is the safe answer.
        return 0

load_settings

load_settings()

The saved settings, or {}; a damaged file reads as empty rather than stopping Taters from starting.

Source code in src\taters\helpers\settings.py
 92
 93
 94
 95
 96
 97
 98
 99
100
def load_settings() -> Dict[str, Any]:
    """The saved settings, or ``{}``; a damaged file reads as empty rather
    than stopping Taters from starting."""
    path = settings_path()
    try:
        doc = json.loads(path.read_text(encoding="utf-8"))
    except (OSError, ValueError, UnicodeDecodeError):
        return {}
    return doc if isinstance(doc, dict) else {}

model_cache_dir

model_cache_dir()

Where downloaded models live; see the module docstring for the order.

Source code in src\taters\helpers\settings.py
159
160
161
def model_cache_dir() -> Path:
    """Where downloaded models live; see the module docstring for the order."""
    return model_cache_source()[0]

model_cache_source

model_cache_source()

The model cache and which rule chose it: "environment" (TATERS_MODEL_CACHE), "setting" (chosen in Settings), "hub environment" (HF_HUB_CACHE/HF_HOME) or "default".

Source code in src\taters\helpers\settings.py
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
def model_cache_source() -> Tuple[Path, str]:
    """
    The model cache and which rule chose it: ``"environment"``
    (``TATERS_MODEL_CACHE``), ``"setting"`` (chosen in Settings),
    ``"hub environment"`` (``HF_HUB_CACHE``/``HF_HOME``) or ``"default"``.
    """
    override = os.environ.get(MODEL_CACHE_ENV)
    if override:
        return Path(os.path.expandvars(override)).expanduser(), "environment"
    saved = load_settings().get(MODEL_CACHE_KEY)
    if saved:
        return Path(os.path.expandvars(str(saved))).expanduser(), "setting"
    if os.environ.get("HF_HUB_CACHE") or os.environ.get("HF_HOME"):
        return _hub_default(), "hub environment"
    return _hub_default(), "default"

save_setting

save_setting(key, value)

Write one setting, keeping the others, atomically.

Source code in src\taters\helpers\settings.py
103
104
105
106
107
108
109
110
111
112
113
def save_setting(key: str, value: Any) -> Path:
    """Write one setting, keeping the others, atomically."""
    from .atomic import atomic_write

    doc = load_settings()
    doc[str(key)] = value
    path = settings_path()
    path.parent.mkdir(parents=True, exist_ok=True)
    with atomic_write(path, mode="w", encoding="utf-8") as fh:
        json.dump(doc, fh, indent=1)
    return path

settings_path

settings_path()

$TATERS_HOME/settings.json (or ~/.taters/settings.json).

Source code in src\taters\helpers\settings.py
87
88
89
def settings_path() -> Path:
    """``$TATERS_HOME/settings.json`` (or ``~/.taters/settings.json``)."""
    return _home() / "settings.json"

taters.helpers.update_check

"A newer version is out" -- said once, quietly, and never in the way.

The rules this is built around, in order of importance:

  1. The menu never waits for the network. The note is read from a cached answer, which is instant; the refresh that produces that answer runs in a background thread and its result is for next launch. A slow network, a proxy that blackholes the request, no network at all -- none of it can delay the first screen or stop Taters starting.
  2. It can be turned off, and turning it off is honored everywhere. Some people run this on data that cannot leave the building, and an unexplained outbound connection is a conversation with IT nobody wants. Setting TATERS_NO_UPDATE_CHECK=1, or the saved setting, stops it dead -- no thread, no request.
  3. It cannot break anything. Every path here is wrapped. A failure leaves no note and no complaint; there is nothing here worth interrupting a run for.
  4. It says one thing. "Newer version available: v0.7.3" under the banner, in dim text. Not an exhortation, not something to dismiss.

note

note()

The line to print under the banner, or "" -- read from cache, instantly.

Returns "" when: the check is off, Taters is running from a source tree with no version metadata, nothing has been cached yet (the first launch), the cached answer is not newer, or anything at all goes wrong.

Source code in src\taters\helpers\update_check.py
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
def note() -> str:
    """The line to print under the banner, or "" -- read from cache, instantly.

    Returns "" when: the check is off, Taters is running from a source tree
    with no version metadata, nothing has been cached yet (the first launch),
    the cached answer is not newer, or anything at all goes wrong.
    """
    try:
        if _disabled():
            return ""
        here = _release(_installed())
        if here is None:
            # no metadata, or a version this cannot compare. running from a
            # checkout is the normal case for that, and a developer does not
            # need telling that PyPI is behind them.
            return ""
        seen = load_settings().get(SETTING_KEY, {}).get("latest", "")
        there = _release(str(seen))
        if there is None or there <= here:
            return ""
        return f"Newer version available: v{seen}"
    except Exception:
        return ""

refresh_in_background

refresh_in_background()

Start the refresh, or don't. Returns the thread for tests to join.

A daemon thread: if somebody quits Taters two seconds after opening it, the interpreter exits without waiting on a socket nobody is reading.

Source code in src\taters\helpers\update_check.py
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
def refresh_in_background() -> Optional[threading.Thread]:
    """Start the refresh, or don't. Returns the thread for tests to join.

    A daemon thread: if somebody quits Taters two seconds after opening it,
    the interpreter exits without waiting on a socket nobody is reading.
    """
    try:
        if _disabled() or _release(_installed()) is None:
            return None
        thread = threading.Thread(target=_fetch_and_store, name="taters-update-check",
                                  daemon=True)
        thread.start()
        return thread
    except Exception:
        return None

taters.helpers.feature_columns

What every feature table calls its columns, declared once and checked at build time.

The problem this solves only shows up when two measures meet. Feature tables get joined side by side into one analysis table, and if two of them use the same column name, something has to give. stats.assemble already handles that: it refuses two tables with the same stem, and renames any column used by two tables to <stem>__<column> in every table that has it, so the outcome does not depend on file order.

That is a good safety net and it stays. But it is reactive, and reactive renaming has a property that is poison for research: the name depends on what else ran. A column is Topic_1 in a pipeline with one topic model and lda_topics__Topic_1 in a pipeline with two. The same instrument, the same corpus, two different column names -- so a script written against one study silently fails against the next, and two results tables cannot be compared without knowing what else was in each run.

So the names we choose have to be disjoint up front. Each feature module declares what it writes; :func:overlaps finds any two declarations that could produce the same name; and tests/test_feature_columns.py fails the build if any do, naming both offenders. Collisions among shipped measures therefore never happen, the reactive rename never fires for them, and Topic_1 is Topic_1 in every pipeline forever.

Three kinds of column name exist, and only two of them can be policed here:

  • Fixed -- names we choose and always write (flesch_reading_ease).
  • Patterned -- names we choose whose tail is a number or a setting (Theme_{n}, msttr_{n}). Declared as a pattern, matched as one.
  • Dynamic -- names that come out of the user's own data: the categories in their dictionary, their archetype names, however many dimensions their encoder has. We cannot know these and do not pretend to. They are declared with a sentence saying where they come from, and they are exactly what the reactive rename in assemble is for.

Adding a measure means adding a FEATURE_COLUMNS to its module. The test finds it through the recipe catalog rather than through a list kept here, so forgetting is a failure rather than a silence.

ColumnSpec dataclass

ColumnSpec(
    label,
    names=(),
    patterns=(),
    reduces_to="Component",
    dynamic="",
    _module="",
)

One module's claim about the columns it writes.

Parameters:

Name Type Description Default
label str

What to call this measure in a collision message. A person reads it.

required
names tuple of str

Column names always written, spelled exactly.

()
patterns tuple of str

Names whose tail varies with a setting, with a placeholder where the varying part goes. {n} is a number ("Theme_{n}", "msttr_{n}") and {*} is anything ("pos_{*}", where the tail is a tag). Exactly one placeholder per pattern; the rest is matched literally.

The distinction earns its keep: sentence embeddings write e0, e1, … and transformer embeddings write e_1, e_2, …, which differ by one character and do not collide -- but a placeholder meaning "anything" would report them as if they did, and the noise would train somebody to ignore this check.

()
reduces_to str

What a PCA over these columns produces, as a singular noun. Reducing a topic model's topics does not give you "components", it gives you something worth naming -- so the topic models say "Supertopic" and everything else keeps "Component". It is declared here rather than worked out by the statistics stage because this is where the codebase already knows what a column is.

'Component'
dynamic str

Non-empty when this module also writes columns whose names come from the user's data rather than from us. The text says where they come from. It is documentation, not an exemption: whatever is declared in names and patterns is still checked. The undeclarable rest is what assemble's reactive rename exists for.

''
Notes

Bookkeeping columns are left out on purpose. Several modules write token_count beside their measures, and declaring it would report a collision on a column no analysis ever treats as a feature -- assemble keeps those aside from the feature sets and renames them harmlessly if two tables carry one. Declare what somebody would analyze.

pattern_regex

pattern_regex(pattern)

A pattern as a regex: everything literal but the one placeholder.

{n} becomes \d+ and {*} becomes .+. Use {n} whenever the varying part really is a number, because it is what tells e{n} and e_{n} apart -- two real patterns in this codebase that would otherwise read as the same one.

Source code in src\taters\helpers\feature_columns.py
129
130
131
132
133
134
135
136
137
138
139
def pattern_regex(pattern: str) -> "re.Pattern[str]":
    """A pattern as a regex: everything literal but the one placeholder.

    ``{n}`` becomes ``\\d+`` and ``{*}`` becomes ``.+``. Use ``{n}`` whenever
    the varying part really is a number, because it is what tells ``e{n}`` and
    ``e_{n}`` apart -- two real patterns in this codebase that would otherwise
    read as the same one.
    """
    placeholder, expansion = ("{n}", r"\d+") if "{n}" in pattern else ("{*}", ".+")
    head, _, tail = pattern.partition(placeholder)
    return re.compile(f"{re.escape(head)}{expansion}{re.escape(tail)}$")

overlaps

overlaps(specs)

Every pair of declarations that could write the same column name.

Returns:

Type Description
list of (label, label, names)

One entry per colliding pair, with the names they collide on. Empty when the declarations are disjoint, which is the only acceptable state for the measures Taters ships.

Source code in src\taters\helpers\feature_columns.py
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
def overlaps(specs: Sequence[ColumnSpec]) -> List[Tuple[str, str, List[str]]]:
    """
    Every pair of declarations that could write the same column name.

    Returns
    -------
    list of (label, label, names)
        One entry per colliding pair, with the names they collide on. Empty
        when the declarations are disjoint, which is the only acceptable state
        for the measures Taters ships.
    """
    found: List[Tuple[str, str, List[str]]] = []
    for i, first in enumerate(specs):
        for second in specs[i + 1:]:
            shared = _collide(first, second)
            if shared:
                found.append((first.label, second.label, shared))
    return found

by_module

by_module(specs)

Declarations keyed by the module that made them, for error messages.

Source code in src\taters\helpers\feature_columns.py
195
196
197
def by_module(specs: Sequence[ColumnSpec]) -> Dict[str, ColumnSpec]:
    """Declarations keyed by the module that made them, for error messages."""
    return {spec._module or spec.label: spec for spec in specs}

registry

registry()

Every declaration, keyed by module. Built once, and never raises.

A module that cannot be imported (an optional dependency is missing) is skipped rather than fatal: the registry is used to name things nicely, and a missing encoder should not stop a run that never wanted one.

Source code in src\taters\helpers\feature_columns.py
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
def registry() -> Dict[str, ColumnSpec]:
    """Every declaration, keyed by module. Built once, and never raises.

    A module that cannot be imported (an optional dependency is missing) is
    skipped rather than fatal: the registry is used to *name* things nicely,
    and a missing encoder should not stop a run that never wanted one.
    """
    if not _REGISTRY:
        import importlib

        for path in DECLARING_MODULES:
            try:
                spec = getattr(importlib.import_module(path), "FEATURE_COLUMNS", None)
            except Exception:                   # pragma: no cover - optional extras
                continue
            if spec is not None:
                _REGISTRY[path] = spec
    return dict(_REGISTRY)

reduced_name

reduced_name(columns, *, set_name='')

What to call a component built from these columns: "Supertopic", or "Component" when nothing more specific is known.

Matched on the column names themselves rather than threaded down from the pipeline, because by the time the statistics stage reduces a feature set, the set is just a list of column names in a table -- whatever produced it is long out of scope.

Source code in src\taters\helpers\feature_columns.py
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
def reduced_name(columns: Sequence[str], *, set_name: str = "") -> str:
    """
    What to call a component built from these columns: "Supertopic", or
    "Component" when nothing more specific is known.

    Matched on the column names themselves rather than threaded down from the
    pipeline, because by the time the statistics stage reduces a feature set,
    the set is just a list of column names in a table -- whatever produced it
    is long out of scope.
    """
    trimmed = []
    for column in columns:
        prefix = f"{set_name}{SEPARATOR}"
        trimmed.append(column[len(prefix):]
                       if set_name and column.startswith(prefix) else column)

    if not trimmed:
        return "Component"

    for spec in registry().values():
        if spec.reduces_to == "Component":
            continue
        literal = set(spec.names)
        checks = [pattern_regex(p) for p in spec.patterns]
        # *every* column has to belong, not just one. a feature set that mixes
        # a topic model with readability indices reduces to something that is
        # genuinely not a supertopic, and the combined "all" set -- which the
        # statistics stage builds by default -- is exactly that mixture. an
        # `any` here named its components supertopics on the strength of one
        # matching column.
        if all(c in literal or any(rx.fullmatch(c) for rx in checks)
               for c in trimmed):
            return spec.reduces_to
    return "Component"