Utilities & Helpers¶
taters.helpers.feature_gather ¶
AggregationPlan
dataclass
¶
AggregationPlan(
group_by,
per_file=True,
stats=("mean", "std"),
exclude_cols=(),
include_regex=None,
exclude_regex=None,
dropna=False,
)
Plan describing how numeric feature columns should be aggregated.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
group_by
|
Sequence[str]
|
One or more column names used as grouping keys (e.g., |
required |
per_file
|
bool
|
If True, include |
True
|
stats
|
Sequence[str]
|
Statistical reductions to compute for each numeric feature column.
|
("mean", "std")
|
exclude_cols
|
Sequence[str]
|
Columns to drop before filtering/selecting numeric features (e.g., timestamps or free text). |
()
|
include_regex
|
str or None
|
Optional regex; if provided, only columns matching this pattern are kept
(after excluding |
None
|
exclude_regex
|
str or None
|
Optional regex; if provided, columns matching this pattern are removed
(after applying |
None
|
dropna
|
bool
|
Whether to drop rows with NA in any of the group-by keys before grouping. The default keeps them, so rows with a missing key land in their own clearly-labeled group instead of vanishing from the output. |
False
|
Notes
This plan is consumed by :func:aggregate_features. Column filtering happens
before numeric selection; only columns that remain and can be coerced to numeric
will be aggregated.
aggregate_features ¶
aggregate_features(
*,
root_dir,
pattern="*.csv",
recursive=True,
delimiter=",",
encoding="utf-8-sig",
add_source_path=False,
plan,
out_csv=None,
overwrite_existing=False,
verbose=True,
on_progress=None
)
Discover files, read, concatenate, and aggregate numeric columns per plan.
This function consolidates tables from a single folder (or one table in
a database), filters columns, coerces candidate features to numeric,
groups by the specified keys, and computes the requested statistics.
Output columns for aggregated features are flattened with the pattern
"{column}__{stat}".
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root_dir
|
PathLike
|
Folder containing per-item CSVs, a single CSV file, or a database
table ( |
required |
pattern
|
str
|
Glob pattern for selecting files. |
"*.csv"
|
recursive
|
bool
|
Recurse into subdirectories when True. |
True
|
delimiter
|
str
|
CSV delimiter. |
","
|
encoding
|
str
|
CSV encoding for read/write. |
"utf-8-sig"
|
add_source_path
|
bool
|
If True, include absolute path in |
False
|
plan
|
AggregationPlan
|
Aggregation configuration (group keys, stats, filters, NA handling). |
required |
out_csv
|
PathLike or None
|
Output path. If None, defaults to
|
None
|
overwrite_existing
|
bool
|
If False and |
False
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the written CSV of aggregated features (the ref, for a database table). |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If no files match the pattern under |
RuntimeError
|
If files were found but none could be read successfully. |
ValueError
|
If required group-by columns are missing,
or if no numeric columns remain after filtering,
or if per-file grouping is requested but the |
Notes
Group keys are preserved as leading columns in the output. The output places
"source" (and optionally "source_path") first when present. The
statistics in :data:RUNNING_STATS are folded in one pass with a few
numbers per group held; any other statistic goes through pandas and
holds the whole table.
Source code in src\taters\helpers\feature_gather.py
774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 | |
feature_gather ¶
feature_gather(
*,
root_dir,
pattern="*.csv",
recursive=True,
delimiter=",",
encoding="utf-8-sig",
add_source_path=False,
aggregate=False,
plan=None,
group_by=None,
per_file=True,
stats=("mean", "std"),
exclude_cols=(),
include_regex=None,
exclude_regex=None,
dropna=False,
out_csv=None,
overwrite_existing=False,
verbose=True,
on_progress=None
)
Single entry point to concatenate or aggregate feature CSVs from one folder.
If aggregate=False, CSVs are concatenated with origin metadata
(see :func:gather_csvs_to_one). If aggregate=True, numeric feature
columns are aggregated per the provided or constructed plan
(see :func:aggregate_features).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root_dir
|
PathLike
|
Folder containing per-item CSVs (or a single CSV file), or a table
in a database ( |
required |
pattern
|
str
|
Glob pattern for selecting CSV files. |
"*.csv"
|
recursive
|
bool
|
Recurse into subdirectories when True. |
True
|
delimiter
|
str
|
CSV delimiter. |
","
|
encoding
|
str
|
CSV encoding. |
"utf-8-sig"
|
add_source_path
|
bool
|
If True, include a |
False
|
aggregate
|
bool
|
Toggle aggregation mode. If False, files are concatenated. |
False
|
plan
|
AggregationPlan or None
|
Explicit plan for aggregation. Required if |
None
|
group_by
|
Sequence[str] or None
|
Quick-plan keys. Used only when |
None
|
per_file
|
bool
|
Quick-plan flag; include |
True
|
stats
|
Sequence[str]
|
Quick-plan statistics to compute per numeric column. |
("mean", "std")
|
exclude_cols
|
Sequence[str]
|
Quick-plan columns to drop before numeric selection. |
()
|
include_regex
|
str or None
|
Quick-plan regex to include feature columns by name. |
None
|
exclude_regex
|
str or None
|
Quick-plan regex to exclude feature columns by name. |
None
|
dropna
|
bool
|
Quick-plan NA handling for group keys. When True, rows whose group key is missing are dropped before aggregating. |
False
|
out_csv
|
PathLike or None
|
Output table. If None, defaults to
|
None
|
overwrite_existing
|
bool
|
If False and |
False
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the resulting CSV (the ref, for a database table). |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
See Also
gather_csvs_to_one : Concatenate CSVs with origin metadata. aggregate_features : Aggregate numeric columns according to a plan.
Source code in src\taters\helpers\feature_gather.py
294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 | |
gather_csvs_to_one ¶
gather_csvs_to_one(
*,
root_dir,
pattern="*.csv",
recursive=True,
delimiter=",",
encoding="utf-8-sig",
add_source_path=False,
out_csv=None,
overwrite_existing=False,
verbose=True,
on_progress=None
)
Concatenate many CSVs into a single CSV with origin metadata.
Each input table is streamed through, a leading "source" column is
inserted (and optionally "source_path"), and rows are appended. The
final table's columns are the union of every input's, in order of first
appearance, with "source" (and, if present, "source_path")
leading.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root_dir
|
PathLike
|
Folder containing CSVs, a single CSV file, or a database table. |
required |
pattern
|
str
|
Glob pattern for selecting files. |
"*.csv"
|
recursive
|
bool
|
Recurse into subdirectories when True. |
True
|
delimiter
|
str
|
CSV delimiter. |
","
|
encoding
|
str
|
CSV encoding for read/write. |
"utf-8-sig"
|
add_source_path
|
bool
|
If True, include absolute path in |
False
|
out_csv
|
PathLike or None
|
Output path. If None, defaults to
|
None
|
overwrite_existing
|
bool
|
If False and |
False
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the written CSV (the ref, for a database table). |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If no files match the pattern under |
RuntimeError
|
If files were found but none could be read successfully. |
Notes
Cells are copied as the text they are. Column order from inputs is preserved after the leading origin columns.
Source code in src\taters\helpers\feature_gather.py
440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 | |
make_plan ¶
make_plan(
*,
group_by,
per_file=True,
stats=("mean", "std"),
exclude_cols=(),
include_regex=None,
exclude_regex=None,
dropna=False
)
Create an :class:AggregationPlan from simple arguments.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
group_by
|
Sequence[str]
|
Grouping key(s) to use (e.g., |
required |
per_file
|
bool
|
If True, group within files by including |
True
|
stats
|
Sequence[str]
|
Statistical reductions to compute per numeric column. |
("mean", "std")
|
exclude_cols
|
Sequence[str]
|
Columns to drop prior to feature selection. |
()
|
include_regex
|
str or None
|
Regex to include feature columns by name. |
None
|
exclude_regex
|
str or None
|
Regex to exclude feature columns by name. |
None
|
dropna
|
bool
|
Drop rows with NA in any group key. Off by default so rows with a missing key are still reported rather than silently discarded. |
False
|
Returns:
| Type | Description |
|---|---|
AggregationPlan
|
A configured plan instance for :func: |
Source code in src\taters\helpers\feature_gather.py
200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 | |
taters.helpers.find_files ¶
find_files ¶
find_files(
root_dir,
*,
file_type="video",
extensions=None,
recursive=True,
follow_symlinks=False,
include_hidden=False,
include_globs=None,
exclude_globs=None,
absolute=True,
sort=True,
ffprobe_verify=False
)
Discover media files under a folder using smart, FFmpeg-friendly filters.
You can either (a) choose a built-in group of extensions via file_type
("audio"|"video"|"image"|"subtitle"|"archive"|"any") or (b) pass an explicit
list of extensions to match. Matching is case-insensitive; dots are optional
(e.g., ".wav" and "wav" are equivalent). Hidden files and directories are
excluded by default.
For audio/video, ffprobe_verify=True additionally checks that at least one
corresponding stream is present (e.g., exclude MP4s with no audio when
file_type="audio"). This is slower but robust when your dataset contains
“container only” files.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root_dir
|
str | PathLike
|
Folder to scan. |
required |
file_type
|
str
|
Built-in group selector. Ignored if |
'video'
|
extensions
|
Optional[Sequence[str]]
|
Explicit extensions to include (e.g., |
None
|
recursive
|
bool
|
Recurse into subfolders. Default: |
True
|
follow_symlinks
|
bool
|
Follow directory symlinks during traversal. Default: |
False
|
include_hidden
|
bool
|
Include dot-files and dot-dirs. Default: |
False
|
include_globs
|
Optional[Sequence[str]]
|
Additional glob filters applied after extension filtering; |
None
|
absolute
|
bool
|
Return absolute paths when |
True
|
sort
|
bool
|
Sort lexicographically (case-insensitive). Default: |
True
|
ffprobe_verify
|
bool
|
For |
False
|
Returns:
| Type | Description |
|---|---|
list[Path]
|
The matched files. |
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
If |
ValueError
|
If |
Examples:
Find all videos (recursive), as absolute paths:
>>> find_files("dataset", file_type="video")
Use explicit extensions and keep paths relative:
>>> find_files("dataset", extensions=[".wav",".flac"], absolute=False)
Only include files matching a glob and exclude temp folders:
>>> find_files("dataset", file_type="audio",
... include_globs=["**/*session*"], exclude_globs=["**/tmp/**"])
Verify playable audio streams exist:
>>> find_files("dataset", file_type="audio", ffprobe_verify=True)
Source code in src\taters\helpers\find_files.py
112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 | |
taters.helpers.text_gather ¶
csv_to_analysis_ready_csv ¶
csv_to_analysis_ready_csv(
*,
csv_path,
out_csv=None,
overwrite_existing=False,
text_cols=None,
id_cols=None,
mode="concat",
group_by=None,
delimiter=None,
encoding=DEFAULT_ENCODING,
joiner=DEFAULT_JOINER,
num_buckets=1024,
max_open_bucket_files=64,
tmp_root=None,
include_id_cols=True,
carry_cols=None,
agg_cols=None,
row_filters=None,
verbose=True,
on_progress=None
)
Stream a (possibly huge) CSV into a compact analysis-ready CSV with a stable schema and optional external grouping.
Output schema
Always writes a header and enforces a consistent column order:
• No grouping:
text_id,text (plus source_col if mode="separate")
• With grouping:
text_id,text,group_count (plus source_col if mode="separate")
carry_cols inserts the named source columns between the identifiers and
text in both shapes.
Where:
- text_id is either the composed ID from id_cols or row_<n> when
id_cols=None.
- mode="concat" joins all text_cols using joiner per row or group.
- mode="separate" emits one row per (row_or_group, text_col) and
fills source_col with the contributing column name.
Grouping at scale
If group_by is provided, the function performs a two-pass external
grouping that does not require presorting:
1) Hash-partition rows to on-disk “bucket” CSVs (bounded writers with LRU).
2) Aggregate each bucket into final rows (concat or separate mode), writing
group_count to record how many pieces contributed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
csv_path
|
PathLike
|
Source CSV with at least the columns in |
required |
out_csv
|
PathLike | None
|
Destination CSV. If |
None
|
overwrite_existing
|
bool
|
If |
False
|
text_cols
|
Sequence[str] | None
|
Text fields to concatenate or emit separately. May be empty or None:
the spreadsheet is then wrangled without any text -- rows are still
combined, counted, summarized and carried -- and the output has no
|
None
|
id_cols
|
Sequence[str] | None
|
Optional columns to compose |
None
|
carry_cols
|
Sequence[str] | None
|
Source columns to copy through to the output untouched. Distinct from
When grouping, a carried column is written only where its value is the same for every row in the group; where rows disagree the cell is blank, because there is no honest single answer and picking the first row's is the kind of quiet wrong answer that survives into a published table. Names in |
None
|
agg_cols
|
Mapping[str, str] | Sequence[str] | None
|
Number columns to summarize per group, e.g. a per-post score becoming
the average score of everything a user wrote. A sequence of names
means "the mean of each"; a mapping picks the statistic per column
from Values that are blank or not parseable as numbers are skipped rather than treated as zero -- a missing score is missing, not 0 -- and a group with no numeric values at all gets a blank cell, for the same NA-is-not-zero reason a disagreeing carried column does. |
None
|
row_filters
|
Sequence[Sequence[object]] | None
|
Which rows to keep, as Only the spreadsheet's own columns, because nothing has been measured yet: a word count is not available here and does not belong here either. What somebody knows at this point is what they collected -- a screener somebody failed, a condition they are not analyzing, an age below the one they meant to study. It matters most when rows are being combined, since a row inside somebody else's joined text cannot be taken out again afterwards; that is why it is applied before the rows are bucketed rather than filtered later. Summaries run over every row of the group, including rows whose
text cells are empty -- Requires |
None
|
mode
|
str
|
|
'concat'
|
group_by
|
Sequence[str] | None
|
Optional list of columns to aggregate by; works on unsorted CSVs. |
None
|
delimiter
|
str | None
|
Parsing/formatting options. If |
None
|
encoding
|
str | None
|
Parsing/formatting options. If |
None
|
joiner
|
str | None
|
Parsing/formatting options. If |
None
|
num_buckets
|
int
|
External grouping controls (partition count, LRU limit, temp root). |
1024
|
max_open_bucket_files
|
int
|
External grouping controls (partition count, LRU limit, temp root). |
1024
|
tmp_root
|
int
|
External grouping controls (partition count, LRU limit, temp root). |
1024
|
include_id_cols
|
bool
|
When not grouping, write the id columns beside |
True
|
on_progress
|
Optional |
None
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the analysis-ready CSV. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If required columns are missing or |
Examples:
Concatenate two text fields per row:
>>> csv_to_analysis_ready_csv(
... csv_path="transcripts.csv",
... text_cols=["prompt","response"],
... id_cols=["speaker"],
... )
Group by speaker and join rows:
>>> csv_to_analysis_ready_csv(
... csv_path="transcripts.csv",
... text_cols=["text"],
... group_by=["speaker"],
... )
Source code in src\taters\helpers\text_gather.py
390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 | |
resolve_analysis_ready ¶
resolve_analysis_ready(
*,
csv_path=None,
txt_dir=None,
analysis_csv=None,
gathered_csv=None,
text_cols=("text",),
id_cols=None,
mode="concat",
group_by=None,
delimiter=",",
encoding="utf-8-sig",
joiner=" ",
num_buckets=64,
max_open_bucket_files=64,
tmp_root=None,
recursive=True,
pattern="*.txt",
id_from="stem",
include_source_path=False,
overwrite_existing=False,
on_progress=None,
workers=0,
carry_cols=None,
verbose=None
)
Accept an analysis-ready table, or gather one -- the analyzers' front door.
Every text analyzer takes its input three ways: a spreadsheet to gather
from, a folder of documents to gather from, or a table already gathered.
The forty lines that told those apart, announced the gather and called
one of the two gatherers with a dozen forwarded settings were copied into
eleven modules, and had begun to drift (one forwarded verbose and
carry_cols, the others did not). This is that block, once. Settings
recording is unaffected: the provenance decorator reads the analyzer's
own bound arguments, not this function's.
carry_cols and verbose are forwarded to the gatherers only when
given, so the analyzers that never passed them behave exactly as before.
Source code in src\taters\helpers\text_gather.py
1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 | |
txt_folder_to_analysis_ready_csv ¶
txt_folder_to_analysis_ready_csv(
*,
root_dir,
out_csv=None,
recursive=False,
pattern=DOCUMENT_PATTERN,
encoding="utf-8",
id_from="stem",
include_source_path=True,
overwrite_existing=False,
verbose=True,
on_progress=None,
workers=0
)
Stream a folder of documents into an analysis-ready CSV with predictable, reproducible IDs.
Documents are .txt, .docx, .doc and .pdf -- text is
extracted per type by :func:taters.helpers.doc_text.read_document_text
(no OCR: a PDF without a machine-readable text layer has zero text). A
document that cannot be read -- corrupt, password-protected, a legacy
.doc with no way to convert it -- is skipped with a warning naming it,
never allowed to take the whole gather down.
For each readable file matching pattern, the emitted row contains:
- text_id: the basename (stem), full filename, or relative path (see
id_from), and
- text: the extracted text.
- source_path: optional column with path relative to root_dir.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
root_dir
|
PathLike
|
Folder containing documents. |
required |
out_csv
|
PathLike | None
|
Destination CSV. If |
None
|
recursive
|
bool
|
Recurse into subfolders. Default: |
False
|
pattern
|
str
|
Glob(s) for matching files; several can be joined with |
DOCUMENT_PATTERN
|
encoding
|
str
|
Decoding for plain-text files. Default: |
'utf-8'
|
workers
|
int
|
Parallel reader processes -- PDF and Word parsing is CPU-bound, and a
big folder reads several times faster in parallel. |
0
|
id_from
|
str
|
How to derive |
'stem'
|
include_source_path
|
bool
|
If |
True
|
overwrite_existing
|
bool
|
If |
False
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the analysis-ready CSV. |
Examples:
>>> txt_folder_to_analysis_ready_csv(root_dir="notes", recursive=True, id_from="path")
Source code in src\taters\helpers\text_gather.py
1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 1232 1233 1234 1235 1236 1237 1238 1239 1240 1241 1242 1243 1244 1245 1246 1247 1248 1249 1250 1251 1252 1253 1254 1255 1256 1257 1258 1259 1260 1261 1262 1263 1264 1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 | |
taters.helpers.reshape ¶
Merge, stack and reshape spreadsheets, without holding any of them in memory.
Four things a person does to a spreadsheet before it is ready to analyze, and that until now meant leaving Taters for R or pandas:
- :func:
merge_spreadsheets-- add the columns of one file onto the rows of another, matching rows by an ID they share (a participant's age and condition onto every one of their trials). - :func:
stack_spreadsheets-- several files one under another (one per session, per wave, per participant), with a column saying which. - :func:
wide_to_long--score_t1, score_t2, score_t3into onescorecolumn plus a column saying which time. - :func:
long_to_wide-- the other way round.
Every one of them has a planner (:func:plan_merge and friends) that reads
the files, writes nothing, and comes back with a report: how many rows match,
which don't, what will collide. The wizard reads that report back in words
before anything is written, and the writers run the same plan again and
refuse the same things, so the API and the app cannot disagree.
Two rules shape everything here. Nothing holds a file's rows: the stacks and the unpivot stream, and the merge and the pivot go through an on-disk sqlite database in a temp folder, the way the n-gram counter already spills. And keys are compared as text, never as floats: a 17-digit ID must never equal its neighbor because they round the same.
Unlike the text gathers, encoding and delimiter default to detect
rather than to utf-8-sig and comma. The files this handles are the messy
ones -- a cp1252 export from Windows, a semicolon file from a European
Excel, a tab-separated "Unicode Text" dump -- and a default that only works
on the tidy ones is the wrong default. The output is always utf-8-sig and
comma-separated, which every other Taters step reads.
ColumnClash
dataclass
¶
ColumnClash(
name, agrees, disagreements, example, output_names
)
A non-key column present in both files.
MergeReport
dataclass
¶
MergeReport(
left,
right,
on,
right_on,
key_match,
relationship,
many_to_many_keys,
many_to_many_examples,
rows_out,
loose_extra_matches,
loose_examples,
clashes,
columns_out,
what="file",
fan_out_examples=(),
)
What a merge would do, before it does it.
refusal ¶
refusal(*, allow_many_to_many=False)
Why the merge must not be written as asked, or None.
Source code in src\taters\helpers\reshape.py
557 558 559 560 561 562 563 564 565 566 567 568 | |
sentences ¶
sentences(how='left')
The report in words, as the wizard reads it back.
Source code in src\taters\helpers\reshape.py
570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 | |
SideSummary
dataclass
¶
SideSummary(
file,
encoding,
delimiter,
rows,
keys,
renamed_headers,
ragged_short,
ragged_long,
blank_keys,
distinct_keys,
duplicated_keys,
rows_matched,
rows_unmatched,
unmatched_keys,
unmatched_examples,
loose_collisions,
loose_collision_examples,
mangled_ids,
key_samples,
)
One file's half of a merge report.
TableFormat
dataclass
¶
TableFormat(path, encoding, delimiter, columns, renamed=())
How a spreadsheet has to be read, worked out from the file itself.
columns are the names as this module will call them: stripped of
surrounding spaces and any stray BOM, a blank header filled in as
column_<n>, a duplicate made unique as name_2. renamed says
which ones changed, so a screen can mention it.
notes ¶
notes()
What a person should know about how the file was read, or [].
Source code in src\taters\helpers\reshape.py
146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | |
detect_format ¶
detect_format(path, *, encoding=None, delimiter=None)
Work out how to read a spreadsheet: its encoding, separator and columns.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str or Path
|
The spreadsheet. |
required |
encoding
|
str
|
Use this instead of detecting. Detection checks for a BOM, then whether the whole file is UTF-8, then falls back to cp1252 and finally latin-1 -- which reads anything, so a file is never refused for its encoding. |
None
|
delimiter
|
str
|
Use this instead of sniffing. Sniffing is done on decoded text, so a UTF-16 file sniffs correctly. |
None
|
Returns:
| Type | Description |
|---|---|
TableFormat
|
|
Raises:
| Type | Description |
|---|---|
FileNotFoundError
|
No such file. |
ValueError
|
The file is empty, has no header row, or does not look like a CSV at all (an .xlsx or .docx is a zip, and its first bytes decode as text full of control characters). |
Source code in src\taters\helpers\reshape.py
229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 | |
long_to_wide ¶
long_to_wide(
*,
csv_path,
id_cols,
names_from,
values_from,
out_csv=None,
overwrite_existing=False,
carry_cols=None,
on_duplicate="refuse",
max_columns=1000,
key_match="trim",
delimiter=None,
encoding=None,
tmp_root=None,
verbose=True,
on_progress=None
)
Turn rows into columns: one row per key, one column per name.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
csv_path
|
str or Path
|
The long spreadsheet. |
required |
id_cols
|
sequence of str
|
The column(s) that say which rows belong together -- one output row per distinct combination, in order of first appearance. |
required |
names_from
|
str
|
The column whose values become column names: a condition, a time point, a question number. |
required |
values_from
|
str or sequence of str
|
The column holding the values that go into them. Several columns
give |
required |
out_csv
|
str or Path
|
Where to write. Defaults to |
None
|
overwrite_existing
|
bool
|
Rebuild the output if it is already there. |
False
|
carry_cols
|
sequence of str
|
Other columns to bring along. Each has to hold one value per key (a blank defers to the other rows); one that varies is refused, because in a pivot that means the key is missing a column. |
None
|
on_duplicate
|
('refuse', 'first', 'last', 'mean')
|
What to do when one key has the same name on several rows. Refused
by default, with examples, since that nearly always means a missing
key column. |
"refuse"
|
max_columns
|
int
|
Refuse to make more generated columns than this. An ID or timestamp
picked as |
1000
|
key_match
|
('exact', 'trim', 'loose')
|
How the key values are compared; see :func: |
"exact"
|
delimiter
|
str
|
Detected when omitted. |
None
|
encoding
|
str
|
Detected when omitted. |
None
|
tmp_root
|
str or Path
|
Where the working database goes meanwhile; removed afterwards. |
None
|
verbose
|
bool
|
Print what was done. |
True
|
on_progress
|
callable
|
Progress callback. |
None
|
Returns:
| Type | Description |
|---|---|
Path
|
|
Notes
Generated columns are bare names (A, B) when there is one value
column and none would collide with an existing column; otherwise every
one is <values_from>_<name>, so the naming is consistent rather than
piecemeal. Names that are all numbers come out in numeric order. A row
with a blank key or a blank name is left out and counted. Missing
combinations are blank.
Source code in src\taters\helpers\reshape.py
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2032 2033 2034 2035 2036 2037 2038 2039 2040 2041 2042 2043 2044 2045 2046 2047 2048 2049 2050 2051 2052 2053 2054 2055 2056 2057 2058 2059 2060 2061 2062 2063 2064 2065 2066 2067 2068 2069 2070 2071 2072 2073 2074 2075 2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 2090 2091 2092 2093 2094 2095 2096 2097 2098 2099 2100 2101 2102 2103 2104 2105 2106 2107 2108 2109 2110 | |
merge_spreadsheets ¶
merge_spreadsheets(
*,
left_csv,
right_csv,
on,
right_on=None,
out_csv=None,
overwrite_existing=False,
how="left",
key_match="trim",
right_cols=None,
allow_many_to_many=False,
left_delimiter=None,
right_delimiter=None,
left_encoding=None,
right_encoding=None,
tmp_root=None,
verbose=True,
on_progress=None
)
Add the columns of one spreadsheet onto the rows of another, matched by key.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
left_csv
|
str or Path
|
The file whose rows you are keeping -- the trials, the posts, the utterances. Its rows come out in their own order. |
required |
right_csv
|
str or Path
|
The file with the columns to add -- one row per participant, say. |
required |
on
|
str or sequence of str
|
The column(s) that identify the same thing in both files. Several names make a composite key (participant and session together). |
required |
right_on
|
str or sequence of str
|
The same key under different names in |
None
|
out_csv
|
str or Path
|
Where to write. Defaults to |
None
|
overwrite_existing
|
bool
|
Rebuild the output if it is already there; otherwise it is returned as it is. |
False
|
how
|
('left', 'inner', 'outer')
|
|
"left"
|
key_match
|
('exact', 'trim', 'loose')
|
How keys are compared; see :func: |
"exact"
|
right_cols
|
sequence of str
|
Only these right columns are brought over. All of them by default. |
None
|
allow_many_to_many
|
bool
|
A key that repeats in both files multiplies rows, and is almost always a sign that one more key column is needed. Refused unless this is set. |
False
|
left_delimiter
|
str
|
The separators. Sniffed per file when omitted. |
None
|
right_delimiter
|
str
|
The separators. Sniffed per file when omitted. |
None
|
left_encoding
|
str
|
The encodings. Detected per file when omitted: UTF-8 (with or without a BOM), UTF-16, then cp1252, then latin-1. |
None
|
right_encoding
|
str
|
The encodings. Detected per file when omitted: UTF-8 (with or without a BOM), UTF-16, then cp1252, then latin-1. |
None
|
tmp_root
|
str or Path
|
Where the working database goes while the merge runs. It is about the size of the right file and is removed afterwards. |
None
|
verbose
|
bool
|
Print the plan's sentences and the row count. |
True
|
on_progress
|
callable
|
Progress callback. |
None
|
Returns:
| Type | Description |
|---|---|
Path
|
|
Notes
A non-key column present in both files is kept once when the two agree
on every matched row (blank on either side counts as agreement, and the
left's blanks are filled from the right); otherwise the right copy is
renamed <right stem>__<column>. A blank key never matches anything.
Rows with no match are counted and the first few are named, so a key
typed as P001 on one side and 1 on the other is found before the
analysis, not after.
Source code in src\taters\helpers\reshape.py
1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 1047 1048 1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 1084 1085 1086 1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 | |
normalize_key ¶
normalize_key(value, mode='trim')
One key cell as it is compared.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
any
|
The cell. |
required |
mode
|
('exact', 'trim', 'loose')
|
|
"exact"
|
Returns:
| Type | Description |
|---|---|
str
|
|
Source code in src\taters\helpers\reshape.py
402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 | |
plan_long_to_wide ¶
plan_long_to_wide(
*,
csv_path,
id_cols,
names_from,
values_from,
carry_cols=None,
on_duplicate="refuse",
max_columns=1000,
key_match="trim",
delimiter=None,
encoding=None,
tmp_root=None,
on_progress=None
)
What :func:long_to_wide would do. Reads the file once.
Source code in src\taters\helpers\reshape.py
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 | |
plan_merge ¶
plan_merge(
*,
left_csv,
right_csv,
on,
right_on=None,
key_match="trim",
right_cols=None,
left_delimiter=None,
right_delimiter=None,
left_encoding=None,
right_encoding=None,
tmp_root=None,
on_progress=None
)
What :func:merge_spreadsheets would do, without writing anything.
Same parameters; see there. Reads both files once each.
Source code in src\taters\helpers\reshape.py
974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 | |
plan_stack ¶
plan_stack(
*,
csv_paths,
source_col="source",
match_columns="exact",
delimiter=None,
encoding=None,
count_rows=True,
on_progress=None
)
What :func:stack_spreadsheets would do. Reads each file once.
count_rows=False skips the pass and reports zero rows per file: the
column plan alone, from the headers, for a screen that wants to settle
how columns match before it reads anything.
Source code in src\taters\helpers\reshape.py
1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 | |
plan_wide_to_long ¶
plan_wide_to_long(
*,
csv_path,
value_cols,
id_cols=None,
names_to="variable",
values_to="value",
strip_prefix="",
drop_blank=False,
delimiter=None,
encoding=None,
on_progress=None
)
What :func:wide_to_long would do. Reads the file once.
Source code in src\taters\helpers\reshape.py
1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 | |
stack_spreadsheets ¶
stack_spreadsheets(
*,
csv_paths,
out_csv=None,
overwrite_existing=False,
source_col="source",
match_columns="exact",
delimiter=None,
encoding=None,
verbose=True,
on_progress=None
)
Put the rows of several spreadsheets one under another.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
csv_paths
|
sequence of str or Path
|
The files, in the order their rows should come out. At least two. A file listed twice is stacked once. |
required |
out_csv
|
str or Path
|
Where to write. Defaults to |
None
|
overwrite_existing
|
bool
|
Rebuild the output if it is already there. |
False
|
source_col
|
str
|
A first column naming the file each row came from (the file's stem,
or its path under the files' common folder when stems repeat). An
empty string leaves it out. If the name is taken, |
"source"
|
match_columns
|
('exact', 'loose')
|
Whether |
"exact"
|
delimiter
|
str
|
Applied to every file; detected per file when omitted. |
None
|
encoding
|
str
|
Applied to every file; detected per file when omitted. |
None
|
verbose
|
bool
|
Print what was stacked. |
True
|
on_progress
|
callable
|
Progress callback. |
None
|
Returns:
| Type | Description |
|---|---|
Path
|
|
Notes
Columns are the union, in order of first appearance, and a cell a file does not have is blank. A file sharing no column with the others is stacked anyway, and the plan says so: that is usually a merge wanted by mistake.
Source code in src\taters\helpers\reshape.py
1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 | |
wide_to_long ¶
wide_to_long(
*,
csv_path,
value_cols,
id_cols=None,
out_csv=None,
overwrite_existing=False,
names_to="variable",
values_to="value",
strip_prefix="",
drop_blank=False,
delimiter=None,
encoding=None,
verbose=True,
on_progress=None
)
Turn several columns into rows: one row per (original row, column).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
csv_path
|
str or Path
|
The wide spreadsheet. |
required |
value_cols
|
sequence of str
|
The columns that hold the same kind of measurement at different
times or in different conditions -- |
required |
id_cols
|
sequence of str
|
The columns that ride along unchanged on every one of those rows.
By default every column not in |
None
|
out_csv
|
str or Path
|
Where to write. Defaults to |
None
|
overwrite_existing
|
bool
|
Rebuild the output if it is already there. |
False
|
names_to
|
str
|
The new column saying which of |
"variable"
|
values_to
|
str
|
The new column holding the value. |
"value"
|
strip_prefix
|
str
|
Removed from the front of every value column's name in |
""
|
drop_blank
|
bool
|
Leave out the rows whose value is empty. Kept by default, so a balanced design stays visibly balanced. |
False
|
delimiter
|
str
|
Detected when omitted. |
None
|
encoding
|
str
|
Detected when omitted. |
None
|
verbose
|
bool
|
Print what was done. |
True
|
on_progress
|
callable
|
Progress callback. |
None
|
Returns:
| Type | Description |
|---|---|
Path
|
|
Notes
Values are copied as they are. When some value columns hold numbers and others hold text, the plan says so, because the one value column will then read as text everywhere.
Source code in src\taters\helpers\reshape.py
1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 | |
taters.helpers.tables ¶
A table is a table: one way to say where one lives, and one way to read or write it, whether that is a CSV file or a table in a database.
Everything in Taters that produces or consumes a table used to take a
Path. A :class:TableRef is the one value that replaces it. For a file
it is path-like -- open(), Path() and os.fspath() all work, so
nothing that still thinks in files has to change -- and for a database
table it is db://<connection>/<database>/<logical/path>, which is not path-like on
purpose: a call site that has not learned about databases fails with a
TypeError the moment it is handed one, instead of quietly creating a
folder called db: and writing a CSV into it.
The canonical string form goes everywhere a path string went: YAML presets,
{{save_as}} artifacts, lists of feature tables, command-line flags. A
trailing .csv on a database address is dropped, so a template written
as {{var:features_dir}}/readability.csv renders either way.
TableRef ¶
TableRef(path=None, *, profile='', database='', logical='')
Where a table lives. A file path, or
db://<connection>/<database>/<logical>.
Only the file kind is path-like. database is the database on that
connection (main for a SQLite file, which only has the one);
logical is the slash-separated name inside it
(taters/study7/features/readability); the physical table name is
derived from it by the database layer.
Source code in src\taters\helpers\tables.py
65 66 67 68 69 70 71 72 73 | |
name
property
¶
name
What to call the table in a sentence: a file's name with its extension, a database table's logical name.
root
property
¶
root
db://<connection>/<database> for a database table: the place
every table of a run shares. A file's folder.
local_dir ¶
local_dir(work_dir=None)
A folder on disk for things that are not rows: a model file, a
figure, a report fragment beside this table. For a file it is the
file's folder; for a database table it is db_local/<connection>/
<database>/<logical parent> under the work folder.
Source code in src\taters\helpers\tables.py
137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 | |
sibling ¶
sibling(stem, suffix='.csv')
Another table beside this one: features/x.csv -> features/y.csv.
Source code in src\taters\helpers\tables.py
131 132 133 134 135 | |
under ¶
under(logical)
Another table in the same database, by its logical name.
Source code in src\taters\helpers\tables.py
127 128 129 | |
append_table ¶
append_table(
ref,
fieldnames,
*,
key="text_id",
encoding="utf-8-sig",
settings=None,
hints=None,
finish=None
)
Open a table for writing that can be picked up where it stopped.
For a step whose rows each cost seconds of a model's time, "retried
rather than resumed" (:func:atomic_write) throws hours away. This
keeps the scratch file instead: rows go to <name>.part with a
flush after every batch, a sidecar <stem>_partial.json records the
settings that produced them, and on the next run the finished keys
are read back so the caller can skip them. The file still appears
under its real name only once complete. A sidecar that disagrees with
the settings, or a header that differs, starts over.
Yields (writer, done): the writer has flush(), and done is
the set of keys already in the scratch file. A database table has no
scratch to keep, so it is written whole through :func:write_table
and done is empty.
finish, when given, is the last pass: called with the header and
the completed rows (a stream of dicts, in order) once every row is in,
it returns the final header and rows, and those become the table --
columns worked out from the whole run, added to it without the rows
ever being held in memory. A file streams its scratch into the final
file; a database table is staged beside the final one and dropped
after, so no text touches the local disk. A finish that fails on
a file leaves the completed rows in the scratch for the next run (a
database table restarts, as it always does).
Source code in src\taters\helpers\tables.py
521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 | |
count ¶
count(ref, *, encoding='utf-8-sig', on_progress=None)
How many rows, by the cheapest honest means.
Source code in src\taters\helpers\tables.py
300 301 302 303 304 305 306 307 308 309 | |
delete_table ¶
delete_table(ref)
Remove a table Taters made, and whatever rides beside it: a file and its sidecars, or a database table with its registry and sidecar rows. A table Taters did not make is left alone. Returns whether anything was removed.
Source code in src\taters\helpers\tables.py
744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 | |
export_table ¶
export_table(
*,
source,
out_csv,
connection="",
columns=None,
overwrite_existing=False,
verbose=True,
on_progress=None
)
Copy a database table, or a join of tables, into a CSV file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
str or mapping
|
|
required |
out_csv
|
str or Path
|
Where the spreadsheet goes. |
required |
connection
|
str
|
The connection a bare table name or a spec belongs to. |
''
|
columns
|
sequence of str
|
Only these columns, in this order. All of them by default. |
None
|
overwrite_existing
|
bool
|
Rebuild the file if it is already there. |
False
|
verbose
|
bool
|
Print what was copied. |
True
|
on_progress
|
callable
|
Progress callback. |
None
|
Returns:
| Type | Description |
|---|---|
Path
|
|
Source code in src\taters\helpers\tables.py
796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 | |
fingerprint ¶
fingerprint(ref, *, encoding='utf-8-sig', delimiter=',')
The table's fingerprint from its rows (see db.Fingerprint).
Source code in src\taters\helpers\tables.py
648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 | |
handle ¶
handle(ref)
What an analyzer returns: the Path for a file (unchanged for every
caller that ever read one), the ref itself for a database table.
Source code in src\taters\helpers\tables.py
737 738 739 740 741 | |
import_table ¶
import_table(
*,
csv_path,
out,
unique_key="",
delimiter=None,
encoding=None,
overwrite_existing=False,
verbose=True,
on_progress=None
)
Copy a spreadsheet into a database table.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
csv_path
|
str or Path
|
The spreadsheet. Its encoding and separator are detected unless given. |
required |
out
|
str
|
|
required |
unique_key
|
str
|
A column that identifies each row; it becomes the table's key, and a repeated value stops the write. |
''
|
delimiter
|
str
|
How to read the file; detected when omitted. |
None
|
encoding
|
str
|
How to read the file; detected when omitted. |
None
|
overwrite_existing
|
bool
|
Replace a table Taters made earlier under this name. |
False
|
verbose
|
bool
|
Print what was copied. |
True
|
on_progress
|
callable
|
Progress callback. |
None
|
Returns:
| Type | Description |
|---|---|
TableRef
|
|
Source code in src\taters\helpers\tables.py
874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 | |
is_ref ¶
is_ref(value)
Whether value could name a table: a path-like or a db address.
Source code in src\taters\helpers\tables.py
180 181 182 183 | |
modified_at ¶
modified_at(ref)
When the table was last written, as a timestamp; None when absent.
Source code in src\taters\helpers\tables.py
312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 | |
output_ref ¶
output_ref(given, *, source, default)
Where an analyzer's table goes: what the caller said, or the default
beside the source. default is the folder path under the run folder
(("features", "readability")); the table takes the source's name.
A database source defaults to the same database, under
taters/<default...>/<name>, so anything in gives anything out only
when somebody asks for it.
Source code in src\taters\helpers\tables.py
720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 | |
parse_ref ¶
parse_ref(value)
A :class:TableRef from a path, a string, or a ref.
Accepts the db:/connection/... spelling too, which is what a
db:// string becomes after one careless Path() on the way.
Source code in src\taters\helpers\tables.py
186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 | |
read_cells ¶
read_cells(
ref,
*,
columns=None,
encoding="utf-8-sig",
delimiter=",",
on_rows=None
)
Every row as a list of text cells, in columns order (the header's
when none are named). The positional twin of :func:read_rows, for the
matrix-shaped readers (a document-term matrix, a PCA) where a dict per
row of five thousand cells is most of the work.
Source code in src\taters\helpers\tables.py
360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 | |
read_rows ¶
read_rows(
ref,
*,
columns=None,
encoding="utf-8-sig",
delimiter=",",
on_rows=None
)
Every row as a dict of text cells, in the table's own order.
Source code in src\taters\helpers\tables.py
331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 | |
read_sidecar ¶
read_sidecar(ref, kind)
<stem>_<kind>.json beside a file, or the database's sidecar row.
Source code in src\taters\helpers\tables.py
674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 | |
read_str_table ¶
read_str_table(ref, *, encoding='utf-8-sig', columns=None)
The whole table as an all-string DataFrame (what the statistics
read), or only columns of it, in the table's own order.
Source code in src\taters\helpers\tables.py
407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 | |
require_local ¶
require_local(ref, what)
The path of a file ref, or a plain refusal for a database one.
Source code in src\taters\helpers\tables.py
783 784 785 786 787 788 789 | |
tables_under ¶
tables_under(root)
Every table Taters wrote under a database root such as
db://lab/study7/taters/study7/; nothing for a folder (its files are
its own). A root that is just the database, db://lab/study7/, lists
every table Taters made in it.
Source code in src\taters\helpers\tables.py
768 769 770 771 772 773 774 775 776 777 778 779 780 | |
write_table ¶
write_table(
ref,
fieldnames,
*,
encoding="utf-8-sig",
hints=None,
on_rows=None,
fingerprint=False,
unique_key=""
)
Open a table for writing; it appears only once complete.
For a file this is exactly atomic_write plus csv.DictWriter --
the bytes are what they always were. For a database table the rows are
loaded, typed and swapped in by :func:taters.helpers.db.write_rows.
hints names columns whose type is known ("text", "int",
"float"). The writer's fingerprint is set on exit when asked
for (a database table always has one; a file's costs a second pass over
the cells, so it is opt-in).
Source code in src\taters\helpers\tables.py
466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 | |
taters.helpers.sources ¶
A database source: one table, or a table with others joined on, as a thing a preset can hold and the wizard can read back before anything is run.
Not SQL text. A :class:Source is a base table plus zero or more joins on
named keys, with the output columns named, and it is rendered into SQL for
whichever engine the connection speaks -- from quoted identifiers only,
every one checked against the catalog, so there is nothing to inject and a
column that has been dropped since the pipeline was made is found before a
step runs. Saved in YAML it reads like this::
source:
from: {table: posts, as: p}
join:
- {table: users, as: u, how: left, on: [[p.user_id, u.id]]}
columns: {id: p.id, body: p.body, age: u.age}
order_by: [p.id]
A raw query: is the escape hatch for people who have SQL and know it is
right; it is wrapped as a subquery and never edited.
:func:join_report is the read-back: for each join, the same counts the
spreadsheet merge computes (matched, unmatched with examples, keys repeating
on both sides, the rows the join would produce), from COUNT queries on
the server, poured into the merge's own :class:MergeReport so the
sentences a person sees are the ones they already know.
Source
dataclass
¶
Source(
connection,
database="",
table="",
alias="",
joins=(),
columns=dict(),
order_by=(),
query="",
schema="",
)
describe ¶
describe()
posts LEFT JOIN users LEFT JOIN outcomes (or a query).
Source code in src\taters\helpers\sources.py
105 106 107 108 109 110 | |
anchor ¶
anchor(spec)
A :class:~taters.helpers.tables.TableRef for the source's base table
(or query): what opens the right session, and what an output
defaults beside. A spec that names no database gets the connection's
saved default, or a refusal when there is none.
Source code in src\taters\helpers\sources.py
169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 | |
coerce ¶
coerce(value, *, connection='')
A :class:Source from a spec, a mapping, a db:// string or a table name.
Source code in src\taters\helpers\sources.py
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 | |
expression_of ¶
expression_of(spec, session, name)
The quoted "alias"."column" behind an output column name.
Source code in src\taters\helpers\sources.py
297 298 299 300 301 302 303 | |
guarded ¶
guarded(spec, session)
The context a source's SQL runs in: read-only when the source is a typed query, nothing special when it is tables Taters rendered itself.
Source code in src\taters\helpers\sources.py
400 401 402 403 404 405 | |
join_report ¶
join_report(spec, session, *, upto=None)
One :class:MergeReport per join: the left side is the base table with
the earlier joins applied, the right side is the table being added.
Source code in src\taters\helpers\sources.py
318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 | |
query_problem ¶
query_problem(query)
Why a typed query cannot be a source, or "" when it reads as one
SELECT. The session it runs on is made read-only as well (see
:func:guarded); this is the cheap first check that says so in words.
Source code in src\taters\helpers\sources.py
383 384 385 386 387 388 389 390 391 392 | |
render ¶
render(spec, session, *, columns=None, order_by_exprs=None)
The SELECT for a source, from quoted identifiers only.
columns narrows the output to those names (in that order).
order_by_exprs replaces the spec's own ordering with ready SQL
expressions (the gather uses it to sort by its group keys). The result
carries no parameters today, but the pair is the shape every caller
handles, so a WHERE pushed down later changes nothing upstream.
Source code in src\taters\helpers\sources.py
259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 | |
resolve_columns ¶
resolve_columns(spec, session)
(output name, alias, column) for every output column, checked
against the catalog. With no columns mapping: every column of every
table, base first; a name present in several tables is kept bare for
the first and prefixed <table>__ after that, merge's rule.
Source code in src\taters\helpers\sources.py
195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 | |
taters.helpers.db ¶
Database connections: where they are, how to log in, what they allow, and how a table gets written into one.
Three engines behind one small seam. SQLite is the standard library and a
real target for a single person's study; MySQL (and MariaDB) and PostgreSQL
are what a lab's data lives in. Each is a :class:Dialect of a few dozen
lines -- how to quote a name, what a placeholder looks like, how to list
tables, how to swap one table for another atomically -- over the DB-API
connection its driver gives. The drivers are imported inside functions, so
importing Taters never imports a database driver, and a missing one is a
sentence naming the extra rather than an import error at startup.
Profiles are the saved part: a name, the engine, where the server is,
which database and which user. Never the password. That comes from
TATERS_DB_PASSWORD_<NAME>, or from a masked prompt once per session,
and lives in this process's memory and nowhere else -- not in the
environment (a child process would inherit it), not in the run log, not in
the manifest, not in settings.json (which gets synced). Pool workers
never connect at all: the parent reads the rows and ships text to them.
Writing is copy-then-swap. Rows load into a scratch table, get their
types, and the finished table replaces the old one in one rename, so a run
killed halfway leaves the previous table exactly as it was. Every table
Taters writes carries a hidden _taters_row so it reads back in the
order it was written, and a row in _taters_tables saying what it is and
a fingerprint of its rows, which is how provenance tells "unchanged" from
"somebody edited this".
DbError ¶
Bases: RuntimeError
Something the database said, in words a person can act on.
Dialect ¶
What differs between engines. The base class is SQLite.
blank ¶
blank(quoted)
True when the cell is NULL or empty after trimming.
Source code in src\taters\helpers\db.py
623 624 625 | |
cast ¶
cast(column, kind)
kind is "int" or "float"; non-finite words become NULL.
Source code in src\taters\helpers\db.py
580 581 582 583 584 585 586 | |
count_tables ¶
count_tables(session, schema='')
How many tables the account can see, in one query -- the probe wants the number, not a description of each.
Source code in src\taters\helpers\db.py
652 653 654 655 656 657 | |
driver_errors ¶
driver_errors()
The exception classes the driver raises for something the server said, so a session can turn them into words. Not a programming error: a TypeError from a bad parameter stays what it is.
Source code in src\taters\helpers\db.py
527 528 529 530 531 532 533 | |
list_databases ¶
list_databases(session)
The databases on the server this account can see; none for a file.
Source code in src\taters\helpers\db.py
648 649 650 | |
numeric ¶
numeric(quoted)
True when a non-blank cell reads as a number. An offer, not a contract: the wizard shows the verdict and the arrows can change it.
Source code in src\taters\helpers\db.py
627 628 629 630 631 632 | |
sort_key ¶
sort_key(quoted)
An expression that puts equal values together, blanks with NULLs.
Source code in src\taters\helpers\db.py
616 617 618 | |
stream_scope ¶
stream_scope(con)
What a streaming read has to run inside: nothing, for most engines.
Source code in src\taters\helpers\db.py
640 641 642 | |
swap ¶
swap(session, name, new, old, schema='')
new becomes name; what was name becomes old. Atomic.
Source code in src\taters\helpers\db.py
588 589 590 591 592 593 594 595 596 597 598 599 600 | |
Fingerprint ¶
Fingerprint(columns)
A table's identity from its rows, independent of their order and of how the engine typed them.
Each row hashes to a digest; the table's fingerprint is a hash over the column names, the row count and the sum of the row digests. A sum rather than an exclusive-or, so two identical rows do not cancel out.
Source code in src\taters\helpers\db.py
413 414 415 416 | |
HostKey
dataclass
¶
HostKey(type, blob, fingerprint)
One key an ssh host presented: its type, the key itself (base64, as a known_hosts line carries it) and the fingerprint people compare.
HostKeyUnknown ¶
HostKeyUnknown(message, profile, keys=())
Bases: DbError
The ssh host is not trusted yet. keys are what it presented
(see :func:host_keys), so the person can be shown them and asked.
Source code in src\taters\helpers\db.py
119 120 121 122 | |
Loader ¶
Loader(
session,
logical,
columns,
*,
hints=None,
on_rows=None,
unique_key=""
)
Rows pushed in one at a time; a typed table swapped in at the end.
add takes a row, finish makes the table and returns its registry
entry, abort drops the scratch. Nothing is buffered beyond one
insert batch, so a table of any size costs the same memory.
hints names columns whose type is known in advance ("text",
"int", "float"); when every column is hinted the rows go
straight into the typed table. Otherwise they load as text and are
copied with casts on the server, so a 500k-row table is never read back
into Python to learn its types. Refuses to replace a table Taters did
not make.
Source code in src\taters\helpers\db.py
2152 2153 2154 2155 2156 2157 2158 2159 2160 2161 2162 2163 2164 2165 2166 2167 2168 2169 2170 2171 2172 2173 2174 2175 2176 2177 2178 2179 2180 2181 2182 2183 2184 2185 2186 2187 2188 2189 2190 2191 2192 2193 2194 2195 2196 2197 2198 2199 2200 2201 2202 2203 2204 2205 2206 2207 2208 2209 2210 2211 2212 2213 2214 2215 2216 2217 2218 2219 2220 2221 2222 2223 2224 2225 | |
LoginHints
dataclass
¶
LoginHints(
user="",
password=None,
socket="",
source="",
os_user="",
problem="",
)
What the ssh host's own database client would do: the login saved
in ~/.my.cnf / ~/.pgpass (user, password), the database's local
socket if there is one, and the ssh user. Nothing here is shown or
saved; a password found is used for the session and forgotten.
Probe
dataclass
¶
Probe(
select=False,
create=False,
drop=False,
registry=False,
version="",
tables=0,
encrypted=None,
notes=(),
)
sentences ¶
sentences(profile)
The test's read-back, one line each.
Source code in src\taters\helpers\db.py
1864 1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878 1879 1880 1881 1882 1883 1884 1885 1886 1887 | |
Profile
dataclass
¶
Profile(
name,
dialect,
host="",
port=0,
user="",
database="",
schema="",
path="",
ssl="required",
ssl_ca="",
password="ask",
export="allowed",
ssh_host="",
ssh_port=0,
ssh_user="",
ssh_key="",
ssh_hostkey="",
socket="",
)
A saved way to reach a database. Everything but the password.
password records how the password is obtained -- "ask" (a
masked prompt, once per session), "env" (TATERS_DB_PASSWORD_<NAME>),
"server" (read from the person's own ~/.my.cnf or ~/.pgpass
on the ssh host, each session) or "none" -- never the password
itself. A password value that is anything else is dropped when the
profile is loaded.
server
property
¶
server
db.lab.edu (with the port when it is not the usual one), or the
socket or address as seen through the ssh tunnel.
where
property
¶
where
study7 on db.lab.edu when pointed at a database, else just the
server; or the file's name.
label ¶
label()
One line for a picker: engine, who at where, and how it logs in. A connection is the server and the login; the database comes later.
Source code in src\taters\helpers\db.py
224 225 226 227 228 229 230 231 232 233 234 235 236 237 | |
Registered
dataclass
¶
Registered(
table_name,
logical,
layout,
columns,
n_rows,
fingerprint,
written_at,
taters_version="",
)
One row of the registry, as Python.
table_columns
property
¶
table_columns
The columns the table really has, besides the row number: every physical column, plus the packed column when there is one.
Session ¶
Session(profile, con, dialect)
One open connection, its dialect, and the small verbs the code uses.
Source code in src\taters\helpers\db.py
1704 1705 1706 1707 | |
read_only ¶
read_only()
Run a block with the session unable to write: a query somebody typed into a YAML file gets exactly the rights a SELECT needs.
Source code in src\taters\helpers\db.py
1771 1772 1773 1774 1775 1776 1777 1778 1779 1780 1781 1782 | |
stream ¶
stream(sql, params=(), batch=_BATCH)
Rows one at a time without holding the result set.
Source code in src\taters\helpers\db.py
1754 1755 1756 1757 1758 1759 1760 1761 1762 1763 1764 1765 1766 1767 1768 1769 | |
accept_host_keys ¶
accept_host_keys(profile, keys, *, save=True)
Trust these keys for this host from now on: OpenSSH's way is a line
per key in ~/.ssh/known_hosts, the same line ssh would have
written after a "yes"; plink's is a fingerprint on the profile, handed
to it as -hostkey every time. Returns the profile to use (changed
only for plink).
Source code in src\taters\helpers\db.py
1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 | |
cell_text ¶
cell_text(value)
A database value as the text a CSV export of the same table would hold.
str() rather than isoformat() for dates, so 2024-01-01
12:00:00 comes out the way a spreadsheet export writes it and an id
built from a query equals one built from a file. Floats use repr,
the shortest text that reads back to the same number.
Source code in src\taters\helpers\db.py
352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 | |
cell_token ¶
cell_token(text)
One cell canonicalized by value, so a CSV and a database copy of the
same rows fingerprint the same whatever the column's type: blank or
non-finite is empty, an integer is i:<n>, another number is
f:<repr(float)>, and anything else is the text itself.
Source code in src\taters\helpers\db.py
376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 | |
connect ¶
connect(profile, password=None, ask=None, on_step=None)
Open a session. Refuses to run inside a pool worker: the parent reads
the rows and ships text, so a worker never needs, or holds, a password.
on_step hears what is being done as it is done -- a tunnel being
opened, a login -- so a screen never just sits there.
Source code in src\taters\helpers\db.py
1805 1806 1807 1808 1809 1810 1811 1812 1813 1814 1815 1816 1817 1818 1819 1820 1821 1822 1823 1824 1825 1826 1827 1828 1829 1830 1831 1832 1833 1834 1835 1836 1837 1838 | |
database_of ¶
database_of(profile)
The database segment of a table address on this profile: main
for a file, else whatever database the profile is pointed at.
Source code in src\taters\helpers\db.py
2539 2540 2541 2542 | |
drop_table ¶
drop_table(session, entry)
Remove a table Taters made, and its registry and sidecar rows.
Source code in src\taters\helpers\db.py
2459 2460 2461 2462 2463 2464 2465 2466 | |
fingerprint_table ¶
fingerprint_table(session, entry)
Recompute a written table's fingerprint from its rows.
Source code in src\taters\helpers\db.py
2451 2452 2453 2454 2455 2456 | |
for_database ¶
for_database(profile, database='')
The profile pointed at one database on its server. main and ""
mean the file itself for SQLite; "" means the saved default otherwise.
Source code in src\taters\helpers\db.py
2529 2530 2531 2532 2533 2534 2535 2536 | |
host_keys ¶
host_keys(profile, log='')
The keys the ssh host presents, by asking it (ssh-keyscan), or from
what plink already printed when it refused: its complaint names the
key's type and fingerprint. Empty when neither is to be had.
Source code in src\taters\helpers\db.py
1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 | |
list_profiles ¶
list_profiles()
Every saved connection, by name. A damaged entry is skipped.
Source code in src\taters\helpers\db.py
260 261 262 263 264 265 266 267 268 269 | |
login_by_hand ¶
login_by_hand(profile)
Run the by-hand login right here, in this terminal, with nothing captured: ssh (or plink) shows the server's key and asks whether to trust it, exactly as it would at a shell prompt, and remembers the answer where it always does. For when the key could not be fetched to show inside the app. Returns the program's exit code.
Source code in src\taters\helpers\db.py
1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 | |
open_tunnel ¶
open_tunnel(profile, timeout=TUNNEL_TIMEOUT)
The local port the database is reachable on through the SSH host, opening the tunnel on first use. A tunnel that cannot be opened says why in ssh's own words, which name the key, the host or the permission that was the problem; one that stalls says how far it got.
Source code in src\taters\helpers\db.py
1265 1266 1267 1268 1269 1270 1271 1272 1273 1274 1275 1276 1277 1278 1279 1280 1281 1282 1283 1284 1285 1286 1287 1288 1289 1290 1291 1292 1293 1294 1295 1296 1297 1298 1299 1300 1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 | |
parse_login_hints ¶
parse_login_hints(text, profile)
The look's output as :class:LoginHints; separated out so it can be
tested without a server.
Source code in src\taters\helpers\db.py
1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 | |
password_env ¶
password_env(name)
lab-db -> TATERS_DB_PASSWORD_LAB_DB.
Source code in src\taters\helpers\db.py
246 247 248 | |
password_for ¶
password_for(profile, ask=None)
The password for a profile, from the environment, then this session's
memory, then ask(question) -- or None when the profile needs none.
ask is the masked prompt; without one (a script, a worker), a
password that is not already known raises and names the variable to
set, so a headless run fails at once rather than at step seven.
Source code in src\taters\helpers\db.py
310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 | |
plink_path ¶
plink_path()
Where plink is: on the PATH, or in PuTTY's usual Windows folders.
Source code in src\taters\helpers\db.py
1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 1044 1045 1046 | |
probe ¶
probe(profile, password=None, ask=None, on_step=None)
Try each thing a run needs and say which worked, by name. on_step
hears each thing as it is tried, so nothing here is a silent wait.
Source code in src\taters\helpers\db.py
1890 1891 1892 1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906 1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920 1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934 1935 1936 1937 1938 1939 | |
query_words ¶
query_words(e, p)
What the server said while a query ran, in words a person can act on. The login explainers cover not getting in; this covers the things that go wrong once in -- the server's disk filling up, the connection dropping, a table the account may not read -- which are almost always somebody else's to fix, and the sentence says whose.
Source code in src\taters\helpers\db.py
1942 1943 1944 1945 1946 1947 1948 1949 1950 1951 1952 1953 1954 1955 1956 1957 1958 1959 1960 1961 1962 1963 1964 1965 1966 1967 1968 1969 1970 1971 1972 1973 1974 1975 1976 1977 1978 1979 | |
read_cells ¶
read_cells(session, entry, *, columns=None, on_rows=None)
The rows of a table Taters wrote, in write order, as lists of text
cells in columns order (the table's own when none are named).
Source code in src\taters\helpers\db.py
2400 2401 2402 2403 2404 2405 2406 2407 2408 2409 2410 2411 2412 2413 2414 2415 2416 2417 2418 2419 2420 2421 2422 2423 2424 2425 2426 2427 2428 2429 2430 2431 2432 2433 2434 2435 2436 2437 2438 2439 | |
read_rows ¶
read_rows(session, entry, *, columns=None, on_rows=None)
The rows of a table Taters wrote, in write order, as dicts of text cells.
Source code in src\taters\helpers\db.py
2442 2443 2444 2445 2446 2447 2448 | |
registered_under ¶
registered_under(session, prefix)
Every registered table whose logical path starts with prefix.
Source code in src\taters\helpers\db.py
2076 2077 2078 2079 2080 2081 2082 2083 2084 2085 2086 2087 2088 2089 | |
remote_command ¶
remote_command(profile, command, timeout=30.0)
Run one shell command on the ssh host, in batch mode, through the
same program and key the tunnel uses. (exit code, stdout, stderr).
Source code in src\taters\helpers\db.py
1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 1131 1132 1133 1134 1135 1136 1137 1138 1139 1140 | |
remote_login_hints ¶
remote_login_hints(profile)
Look on the ssh host for how its own database client logs in, with the
person's consent already given: their ~/.my.cnf (MySQL) or
~/.pgpass (PostgreSQL), and whether the database's socket is there.
A failure to log in comes back as problem, never as an exception,
because this is a look, not the connection.
Source code in src\taters\helpers\db.py
1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 | |
sanitize_table_name ¶
sanitize_table_name(logical)
taters/study7/features/readability -> taters__study7__features__readability.
Lowercase letters, digits and underscores; past 48 characters the name is shortened deterministically with a hash, never by chance.
Source code in src\taters\helpers\db.py
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 | |
save_profile ¶
save_profile(profile)
Write a profile, keeping the others. Never a password.
Source code in src\taters\helpers\db.py
280 281 282 283 284 285 | |
session_for ¶
session_for(name, database='', ask=None)
A session by connection name, on one of its databases; the usual entry from the table layer. A connection is a server and a login; which database to use on it is chosen each time, so it is passed here. Empty means the connection's own default (a SQLite file has only the one).
Source code in src\taters\helpers\db.py
2517 2518 2519 2520 2521 2522 2523 2524 2525 2526 | |
ssh_hint ¶
ssh_hint(p)
The command a person would type to log in to this host by hand:
ssh -p 2200 me@host, or with PuTTY plink -P 2200 -i key.ppk me@host
-- the port goes on a flag, not after a colon.
Source code in src\taters\helpers\db.py
1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 | |
stream_query ¶
stream_query(session, sql, params=(), *, on_rows=None)
A query's column names and its rows as text cells, streamed.
Source code in src\taters\helpers\db.py
2499 2500 2501 2502 2503 2504 2505 2506 2507 2508 2509 2510 2511 2512 2513 2514 | |
tunnel_command ¶
tunnel_command(profile, local_port)
The invocation that forwards local_port to the database as the SSH
host sees it: OpenSSH's ssh, or PuTTY's plink for a .ppk
key. Batch mode either way -- a key, an agent or Pageant, never a
prompt, since there is no terminal to answer one in the middle of a run.
Source code in src\taters\helpers\db.py
1049 1050 1051 1052 1053 1054 1055 1056 1057 1058 1059 1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 1073 1074 1075 1076 1077 1078 1079 1080 1081 1082 1083 | |
uses_putty ¶
uses_putty(profile)
A .ppk key is PuTTY's format, which OpenSSH cannot read; the
tunnel is then opened with PuTTY's own command-line client, plink,
which also uses Pageant and the host keys PuTTY has already accepted.
Source code in src\taters\helpers\db.py
1027 1028 1029 1030 1031 | |
write_rows ¶
write_rows(
session,
logical,
columns,
rows,
*,
hints=None,
on_rows=None
)
Load rows into the table named by logical; see :class:Loader.
Source code in src\taters\helpers\db.py
2344 2345 2346 2347 2348 2349 2350 2351 2352 2353 2354 2355 2356 | |
taters.helpers.feature_average ¶
Average a feature table up to a coarser row.
The other way round from :mod:text_gather. That one joins the texts that
belong together and measures the result once; this one measures every row on
its own and then averages the numbers. They are different analyses, not two
routes to one answer -- joining a speaker's words before measuring versus
averaging their per-utterance scores differ by around 35% on vocabulary
measures -- so which one happened is the user's choice and is recorded.
One table in, one table out, because a feature table's record has to sit
beside it: :func:taters.stats.assemble.assemble_analysis_table reads the
record of every table it joins to work out which columns are bookkeeping, and
:mod:taters.score_model reads it again to check a saved model against the
features it was fitted on. A step that wrote a folder full of tables would
leave all of them unrecorded.
Averaging repeats. Utterances to speakers to conversations is two calls, the second reading the first's output, and it is not the same as one call straight to conversations: the two-step version weights speakers equally, the one-step version weights whoever talked most. That is the reason this takes a table rather than a corpus -- so the chain is visible in the pipeline instead of hidden in an argument.
Nothing here renames a column. A model is fitted on column names and has to be
applicable to text that arrives at any shape, so flesch_reading_ease stays
flesch_reading_ease whether it describes one utterance or a conversation's
mean of means. The pipeline and its manifest are what record the difference;
see the defines_text note in :mod:taters.helpers.provenance.
average_feature_table ¶
average_feature_table(
*,
in_csv,
out_csv,
group_by,
keys_csv=None,
key_cols=("text_id",),
split_col="",
min_rows=1,
overwrite_existing=False,
delimiter=",",
encoding="utf-8-sig",
on_progress=None,
verbose=True
)
One row per group, each number the mean of the rows that went into it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
in_csv
|
str or Path
|
The feature table to average: one row per text, a row key, and numeric
columns. Non-numeric columns are dropped, since there is no mean of a
word. A file, or |
required |
out_csv
|
str or Path
|
Where to write the averaged table: a file, or a database table. |
required |
group_by
|
sequence of str
|
The columns whose shared values make a group. They come from
|
required |
keys_csv
|
str or Path
|
A table saying which row belongs to which group: the row key, and the grouping columns. Needed for the first average of a feature table, whose only non-numeric column is its key. Omit it once the input already carries the grouping columns. |
None
|
key_cols
|
sequence of str
|
What identifies a row in both tables, and the name the averaged row
key is written under. The averaged key is the group's values joined
with |
``("text_id",)``
|
split_col
|
str
|
A column that splits a row legitimately and must survive the average
rather than be grouped away -- |
""
|
min_rows
|
int
|
Leave out groups assembled from fewer rows than this. |
1
|
overwrite_existing
|
bool
|
Rebuild the table if it is already there. |
False
|
delimiter
|
str
|
How to read the CSVs. The output is written comma-separated, as every feature table is. |
','
|
encoding
|
str
|
How to read the CSVs. The output is written comma-separated, as every feature table is. |
','
|
on_progress
|
callable
|
Progress callback. |
None
|
verbose
|
bool
|
Print a line about what was averaged. |
True
|
Returns:
| Type | Description |
|---|---|
Path or TableRef
|
|
Notes
A group whose values are all blank gets the row key "group", matching
:func:taters.helpers.text_gather.csv_to_analysis_ready_csv. A blank cell
is skipped rather than counted as zero, so a column measured on only some
of a speaker's turns is the mean of the turns it was measured on; the
count column reports the rows in the group, not the rows behind any one
number.
Source code in src\taters\helpers\feature_average.py
109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 | |
taters.helpers.library ¶
The library: user-imported assets that outlive any one project.
A "dictionary" here is the general term -- a content-coding dictionary and an
archetype dictionary are different kinds, with different formats, consumed by
different modules. The library stores each kind in its own folder under the
user's home, so importing a dictionary once makes it available to every
pipeline on the machine, wherever it is run from. That replaces the old
convention of a dictionaries/ folder relative to the working directory,
which silently failed for anyone who did not happen to have one.
No UI imports (the :mod:taters.helpers.gpu pattern): the wizard is one
consumer, but the runner or a future GUI can read the same library. All the
operations are deliberately file-level and boring -- an entry is its file,
its name is the filename stem -- so a user can also just manage the folder by
hand and nothing here will be surprised.
Adding a new kind -- pretrained classifier models are the expected next one --
is a :data:KINDS entry plus a library= line on the recipe that consumes
it; the manager, the picker, and the empty-library contingency in the wizard
all key off those two declarations.
LibraryCollision ¶
LibraryCollision(existing)
Bases: Exception
An import would overwrite an entry that already exists.
Raised instead of overwriting so the UI can ask replace-or-rename; the
existing path rides along as .existing.
Source code in src\taters\helpers\library.py
374 375 376 | |
LibraryKind
dataclass
¶
LibraryKind(
id,
label,
help,
suffixes,
deep_check=None,
describe_entry=None,
)
One category of importable asset.
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
Stable identifier; also the folder name under the library. |
label, help |
str
|
What menus call it, and one sentence including the accepted formats. |
suffixes |
tuple of str
|
File extensions this kind accepts, lowercase with the dot. |
asset_problem ¶
asset_problem(path, kind=None)
Why this file cannot work as a library asset, or "" when it can.
The canonical check, shared by import (refuse the file while the user is holding it and can act) and by the analyzers (a file can still arrive by CLI path without ever passing through the library). Two layers:
- cheap shape heuristics, always: an empty file is not a dictionary, and a
.dicwith no%...%category header is a bare word list -- the mistake that made contentcoder die with a bare "list index out of range" mid-run, an hour after it was made; - with a
kind, that kind's :attr:~LibraryKind.deep_check-- the real parser. Import passes the kind; the analyzers do not, because they construct the real parser on the very next line and asking it twice buys nothing.
Source code in src\taters\helpers\library.py
793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 | |
delete ¶
delete(kind, name)
Remove an entry and the weights it carries. Permanent, so the UI confirms before calling this.
Source code in src\taters\helpers\library.py
966 967 968 969 970 971 972 973 | |
display_name ¶
display_name(path)
How an entry is named in menus: the filename stem.
Source code in src\taters\helpers\library.py
574 575 576 | |
entries ¶
entries(kind)
Every entry of this kind, sorted by name. Only the kind's own formats count -- a stray .txt dropped into the folder by hand is ignored, not an error.
Source code in src\taters\helpers\library.py
560 561 562 563 564 565 566 567 568 569 570 571 | |
expand ¶
expand(kind, values)
Resolve a mixed list of files and folders to this kind's files.
A folder means "everything of this kind inside it, recursively" -- the reading the analyzers give a folder path, and what the wizard's default writes into a preset (the kind's whole library folder). Any screen seeded through this shows exactly what a run would use; seeding from the raw values made the picker intersect a folder path against entry file paths and open with everything unticked while the settings row said "all 10".
Source code in src\taters\helpers\library.py
579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 | |
export_to ¶
export_to(kind, name, dest_dir, *, replace=False)
Copy an entry out, keeping its filename. Returns the new path.
Raises:
| Type | Description |
|---|---|
LibraryCollision
|
When the destination file already exists and |
Source code in src\taters\helpers\library.py
882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 | |
find_payload ¶
find_payload(kind, name, digests)
A payload in the library whose files match the digests a manifest records.
A model carried into another run travels as its manifest text only (see
pipelines.run_pipeline._materialize_assets); the weights stay in the
library. The manifest records payload_digests -- sha256 per file --
so the loader can find them here and prove they are the same bytes. The
match is by content, not by name: a renamed library entry still counts.
name is the payload's suffix-bearing name, used to skip payloads of
the wrong shape quickly.
Source code in src\taters\helpers\library.py
750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 | |
import_into ¶
import_into(kind, src, *, replace=False)
Copy a file into the library.
Raises:
| Type | Description |
|---|---|
ValueError
|
For a format the kind does not accept -- with the formats it does,
because "wrong extension" without the right ones is a dead end -- or
for a file the kind's own parser cannot load (see
:func: |
LibraryCollision
|
When an entry with this name exists and |
Source code in src\taters\helpers\library.py
828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 | |
kind_by_id ¶
kind_by_id(kind_id)
One kind, or a KeyError that names the valid ids (it is always a typo).
Source code in src\taters\helpers\library.py
379 380 381 382 383 384 385 386 | |
kind_dir ¶
kind_dir(kind)
This kind's folder, created and kept level with what the package ships.
Seeding used to happen only when the folder did not exist, which meant a
release that added a dictionary reached new users and nobody else. So it
now reconciles file by file against a ledger of what we have installed
before (see :func:_seed), and an upgrade brings its new and corrected
built-ins to a library that has been in use for years.
What it will never do is undo a decision: a built-in the user deleted stays deleted, and a copy they edited stays edited. An empty library is a valid state the UI explains, not one this quietly "repairs."
Source code in src\taters\helpers\library.py
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 | |
library_home ¶
library_home()
Where the library lives: $TATERS_HOME/library, or ~/.taters/library.
The environment override exists so tests -- and anyone who wants their library on a different drive -- can move the whole thing without patching.
Source code in src\taters\helpers\library.py
389 390 391 392 393 394 395 396 397 | |
model_files ¶
model_files(folder)
The .json manifests under a folder, without the ones inside a payload.
What a step handed "a folder" resolves to, and what the finish screen
scans a whole run folder with. Only .json files are walked -- a run
over thousands of recordings holds tens of thousands of WAVs and CSVs,
and listing every one of them to find a handful of manifests made the
finish screen wait on a network drive. A payload declared by any
manifest found is pruned, wherever in the tree it sits.
Source code in src\taters\helpers\library.py
674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 | |
payload_dirs ¶
payload_dirs(folder)
Every payload path declared by the manifests directly in folder.
Source code in src\taters\helpers\library.py
641 642 643 644 645 646 647 648 649 | |
payload_of ¶
payload_of(entry)
The sibling files and folders a .json entry declares as its payload.
Empty for anything that is not a JSON manifest with a payload list.
Names are taken as siblings only: a name with a path separator, or a
dot entry, is ignored rather than allowed to point outside the folder.
Source code in src\taters\helpers\library.py
614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 | |
payload_size ¶
payload_size(entry)
Bytes of an entry and everything it carries.
Source code in src\taters\helpers\library.py
736 737 738 739 740 741 742 743 744 745 746 747 | |
rename ¶
rename(kind, old, new)
Rename an entry, keeping its suffix.
The suffix carries the format, which renaming must not be able to lie about -- so a new name arriving with an extension has it stripped rather than honored.
Source code in src\taters\helpers\library.py
909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 | |
taters.helpers.model_spec ¶
One description of a saved model, whatever kind it is.
Taters writes several kinds of reusable instrument -- a MEM topic model, a ridge regression, a classifier -- and they were each self-describing to their own scoring function and to nothing else. That was fine while each kind had its own menu entry, and stops being fine the moment a screen wants to offer models as a category: to list them it has to say what each one is, to score with one it has to know whether the thing needs text or features, and to write the results it has to know what columns will appear.
So each kind registers those three answers here, and every screen asks this module instead of sniffing JSON keys for itself. The alternative -- a per-kind branch at each call site -- is the shape where adding a fourth kind means finding all of them.
Adding a kind of model
Everything a screen or a scoring run needs to know about a model comes from this registry, so a new kind -- a fine-tuned transformer, a gradient-boosted tree, whatever comes next -- is added in one place:
- Give it a
kindtag in the file it writes ("taters-<x>-model") and aformatinteger. -
Add a :class:
ModelTypeto :data:MODEL_TYPESstating: -
needs--"text"if it reads text and re-derives its own instrument (the MEM pattern),"features"if it reads feature columns someone else measured. That one word decides whether the settings-provenance gate applies to it, and nothing else has to know. read-- doc ->(outputs, columns, inputs), so listings can say what it is and what it will add.score--"module:function"for applying it. Omit this and scoring refuses by name instead of guessing.- a
"payload"list in the model file -- the sibling weights the library moves with it (seehelpers.library.payload_of) write_outputs-- if its output columns can be renamed at import.classes-- doc ->{outcome: [class, ...]}when it predicts categories, so a class can be relabeled per model ("0"->"control") and every applier writes the label through :func:class_label.-
apply_settings-- the defaults that govern how it is applied to new data (batch size, weighting, whether to emit probabilities). They live in the model file'sapplyblock, editable per model in Settings, and an applier reads them through :func:apply_defaultsso a pipeline gets the model's own settings unless a call overrides them. -
Add a
_load_model-style gate in the module that owns it, and register it in :func:taters.helpers.library._model_problemso the import screen is exactly as strict as run time.
Nothing else needs editing. In particular the scoring entry point, the library folder, the naming flow, the wizard row and the provenance gate are all written against this registry rather than against the three kinds that happen to exist today.
Naming
A model file is named by whoever exported it, which is nearly always the
step's own output name (ridge__all__age.json), and that says nothing about
what it predicts. So a model carries a name the researcher chose, and
output names for the columns it will write: a ridge fitted to predict
age on a blog corpus should be able to land in a new dataset as
pred_age_blogs, not as pred_age colliding with the age already
there. Both are stored in the model file, so they travel with it.
ApplySetting
dataclass
¶
ApplySetting(
default, help, kind="str", choices=None, validate=None
)
One default that governs how a kind of model is applied to new data.
Stored per model in the file's apply block and read back through
:func:apply_defaults, so a model carries its own settings wherever it
goes -- a word-vector model that should weight by types, a predictor
that fits a small card at batch 8 -- rather than every pipeline having
to know. kind says how a typed-in value is read: "int",
"float", "bool", "str", or "text" for free text that is
not one of a fixed set. choices restricts a str to a list.
coerce ¶
coerce(value, name)
The value as this setting stores it, or a refusal naming both.
Source code in src\taters\helpers\model_spec.py
230 231 232 233 234 235 | |
FeaturePlan
dataclass
¶
FeaturePlan(
model,
slug,
tables=(),
problems=(),
path="",
controls=(),
)
Everything a run needs in order to score with one model, as plain data.
Deliberately free of paths, recipes and pipeline templates: it is built from the model file alone, so the composer stays pure and a test can hand a synthetic plan straight in.
problems is the honest half. A plan with problems cannot be replayed,
and each entry is a sentence a researcher can act on -- which is better
than a partial replay, because a partial replay is a wrong answer.
ModelInfo
dataclass
¶
ModelInfo(
path,
type_id,
type_label,
name,
outputs,
columns,
inputs,
needs,
needs_tables=(),
provenance=dict(),
library_kind=LIBRARY_KIND,
bulk_outputs=False,
controls=(),
zero_when_absent=(),
classes=dict(),
class_names=dict(),
apply=dict(),
modality="text",
)
What one saved model is, in the terms a screen needs.
Attributes:
| Name | Type | Description |
|---|---|---|
path |
Path
|
The file itself. |
type_id |
str
|
Short kind key: |
type_label |
str
|
What to show a person: |
name |
str
|
The researcher's name for the model, defaulting to the file stem. |
outputs |
tuple of str
|
The output labels, after any renaming -- one per predicted outcome, or one per theme. |
columns |
tuple of str
|
The column names those outputs will actually produce, which is not the same list: a classifier writes a predicted class and a probability per class for one outcome. |
inputs |
tuple of str
|
Feature columns the model needs by name. Empty when the model works from text. |
needs |
str
|
|
needs_tables |
tuple of str
|
Which feature tables, named by the step that writes them. A model records its predictors by name, which is enough to score a table that already has them and no help at all in getting there: someone who fitted on cohesion features and came back a week later had 165 column names and nothing saying a cohesion step was needed. |
library_kind |
str
|
The :data: |
bulk_outputs |
bool
|
True when the outputs are a numbered family (MEM themes) rather than a handful of named quantities, so renaming them means choosing one prefix rather than editing a hundred labels. |
n_outputs |
int
|
How many outputs there are, for a listing that does not want to print a hundred theme names. |
display ¶
display()
age_blogs [ridge] -- the one-line form every menu uses.
Source code in src\taters\helpers\model_spec.py
203 204 205 | |
ModelType
dataclass
¶
ModelType(
id,
label,
kind_tag,
needs,
read,
score=None,
write_outputs=None,
bulk_outputs=False,
classes=None,
apply_settings=dict(),
modality="text",
)
One kind of model, and how to read and rewrite its names.
TablePlan
dataclass
¶
TablePlan(
stem,
target,
instrument,
assets,
digest,
grain,
replay=None,
)
One feature table a model needs, and exactly how to measure it.
Attributes:
| Name | Type | Description |
|---|---|---|
stem |
str
|
The table's name at fit time. Load-bearing: the stem is what fixed the model's predictor names, so a replayed table has to be written under it or the names stop matching. |
target |
str
|
|
instrument |
dict
|
The measuring settings, as literal values. Complete, because they were recorded after defaults were applied -- which is the whole reason a replay is constructible rather than a guess. |
assets |
dict
|
|
digest |
str
|
The instrument digest. Belongs in the private output directory: two models needing identical settings then share one extraction for free, and a re-fit under the same model name cannot short-circuit onto the previous private table. |
grain |
dict
|
What one row was at fit time. Informational only -- never compared. |
replay |
dict or None
|
|
UnknownModel ¶
UnknownModel(message, kind=None)
Bases: Exception
The file is not a saved Taters model, or is a kind this build cannot
describe. Carries the kind tag it did have, when it had one.
Source code in src\taters\helpers\model_spec.py
102 103 104 | |
apply_defaults ¶
apply_defaults(doc)
How a model wants to be applied: registry defaults under its own.
Only the settings its kind registers are read, each coerced to its
type; a key the file carries from a newer Taters is ignored rather than
passed on. An applier takes apply_defaults(doc)[key] wherever its
caller left the argument as None, so the model's own settings win over
the function's signature but never over an explicit call.
Source code in src\taters\helpers\model_spec.py
332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 | |
class_label ¶
class_label(doc, outcome, cls)
The label written for one predicted class: its relabel, or itself.
A model fitted on a condition column coded 0/1 predicts "0" and
"1" -- correct, and unreadable in a results table a month later.
The relabel lives in the model file (class_names), keyed by the
class as fitted, and every applier writes predictions and the
p_<outcome>_<class> columns through this one function, so a model
relabeled in Settings is relabeled however it is invoked.
Source code in src\taters\helpers\model_spec.py
316 317 318 319 320 321 322 323 324 325 326 327 328 329 | |
describe ¶
describe(model_json)
Describe one saved model, or say why it cannot be described.
Deliberately cheap and structural: it reads the file's own declaration of what it is and does not vet the matrices. Vetting is the scoring loader's job (and the library import gate delegates to it), because a listing that had to fully validate every model would be slow and would hide a damaged model behind a blank row instead of naming it.
Source code in src\taters\helpers\model_spec.py
670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 | |
describe_all ¶
describe_all(paths)
Describe every model that can be described, skipping the rest silently.
Used by listings, where one unreadable file in a library folder must not take the whole menu down -- the import gate already refused anything broken, so a file that fails here arrived some other way.
Source code in src\taters\helpers\model_spec.py
714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 | |
describe_encoder ¶
describe_encoder(model_json)
One encoder's row: its name and where it came from, then what training did.
Source code in src\taters\helpers\model_spec.py
1133 1134 1135 1136 1137 1138 1139 1140 1141 1142 1143 1144 1145 1146 1147 1148 1149 1150 1151 1152 1153 1154 | |
edit_model ¶
edit_model(
model_json,
*,
name=None,
outputs=None,
prefix=None,
class_names=None,
apply=None
)
Change what a model is called, what it writes, and how it is applied.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_json
|
PathLike
|
The model file, rewritten in place (atomically). |
required |
name
|
Optional[str]
|
The researcher's name for the model. Shown in every menu. |
None
|
outputs
|
Optional[Sequence[str]]
|
One new label per output, in the order :func: |
None
|
prefix
|
Optional[str]
|
For a model whose outputs are a numbered family (MEM themes), the
stem to number from: |
None
|
class_names
|
Optional[Mapping[str, Mapping[str, str]]]
|
|
None
|
apply
|
Optional[Mapping[str, object]]
|
Values for the settings its kind registers (:attr: |
None
|
Returns:
| Type | Description |
|---|---|
ModelInfo
|
The model as it now reads. |
Notes
Every edit rewrites the file rather than a sidecar, so a model that is copied, zipped or emailed keeps the names and settings it was given -- a sidecar would be left behind by every one of those, and the columns would quietly revert.
Source code in src\taters\helpers\model_spec.py
731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 | |
encoder_problem ¶
encoder_problem(model_json)
Why this file is not a usable text encoder, or "" when it is.
Structural and torch-free: the manifest's kind and format, and a payload folder beside it holding a config, weights and a tokenizer. Loading the weights is the extractor's job, minutes later, on the device it chose.
Source code in src\taters\helpers\model_spec.py
1087 1088 1089 1090 1091 1092 1093 1094 1095 1096 1097 1098 1099 1100 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1111 1112 1113 1114 1115 1116 1117 1118 1119 1120 1121 1122 1123 1124 1125 1126 1127 1128 1129 1130 | |
feature_plan ¶
feature_plan(info)
Read a model's own account of the features it was fitted on.
Returns a :class:FeaturePlan. A model that reads text needs no features
and gets an empty plan with no problems -- it re-derives its own
instrument, so there is nothing for a run to arrange.
Source code in src\taters\helpers\model_spec.py
974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1032 1033 1034 1035 1036 | |
library_kind_for ¶
library_kind_for(model_json)
Which library kind a model file belongs in: encoders for an
adapted encoder, models for anything that scores.
Source code in src\taters\helpers\model_spec.py
1185 1186 1187 1188 1189 1190 1191 1192 | |
models_produced ¶
models_produced(folder)
The model files a run left behind, each with a label for a menu.
Scoring models of every registered kind and adapted encoders alike, found by their manifests (never inside a payload folder), so the finish screen and the training task can offer to add them to the library.
Source code in src\taters\helpers\model_spec.py
1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 | |
one_model_path ¶
one_model_path(model_json)
Exactly one model file, or an actionable refusal.
The library hands steps a folder (or a list) meaning "everything of this
kind", which is right for dictionaries and wrong for a model: scoring
with an unspecified one of three is not a thing anyone means. A single
file passes through, a folder or a list resolves to its .json files,
and anything other than exactly one refuses with the fix named.
Owned here because it is about model files, not about any one kind of model: ridge and the topic model each carried a copy, with different refusal wording and one of them letting a folder through unresolved.
Source code in src\taters\helpers\model_spec.py
1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 1223 1224 1225 1226 1227 1228 1229 1230 1231 | |
output_label ¶
output_label(doc, outcome)
The column-name stem for one outcome: its rename, or the outcome itself.
Read by the scoring functions rather than by the wizard, so a renamed model produces renamed columns however it is invoked -- through the app, through the API, or from the command line.
Source code in src\taters\helpers\model_spec.py
303 304 305 306 307 308 309 310 311 312 313 | |
rename_model ¶
rename_model(
model_json, *, name=None, outputs=None, prefix=None
)
Name a model and its output columns -- :func:edit_model without
the class labels and apply settings. Kept for the callers that only
ever name things.
Source code in src\taters\helpers\model_spec.py
859 860 861 862 863 864 865 | |
scorer ¶
scorer(type_id)
The function that scores a new table with this kind of model.
Resolved from the registry rather than chosen at the call site, so that
adding a kind of model is adding a registry entry -- and forgetting the
entry is this refusal, rather than the model being quietly scored by
whichever applier happened to be the else branch.
Source code in src\taters\helpers\model_spec.py
635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 | |
slug ¶
slug(name, fallback='model')
Filesystem- and template-safe: artifact references split on . and
:, and a save_as built from this must not contain either.
One spelling for every model-shaped file name. Three private copies had grown -- here, in ridge (fallback "set") and in score_model -- and two of them kept the dot the third refused, so the same set name could produce two different file names depending on which module wrote it.
Source code in src\taters\helpers\model_spec.py
1060 1061 1062 1063 1064 1065 1066 1067 1068 1069 1070 1071 1072 | |
taters.helpers.settings ¶
Persistent user settings, and the one that matters today: where downloaded models are kept.
Taters has a home folder ($TATERS_HOME or ~/.taters) for the
library; this module puts a small settings.json beside it for choices
that should outlive a session and are not about any one pipeline. The first
such choice is the model cache: every transformer, sentence-transformers
and Whisper model is downloaded by the Hugging Face hub library into its
cache, which lives under the user's home by default. On a shared server
that is the wrong place -- one copy per user of a 500 MB encoder, on a home
partition that is small on purpose -- and the fix has been an environment
variable that has to be set in every shell. Here it is a setting, chosen
once in Settings and applied every time Taters starts, before any of those
libraries is imported.
Resolution order for the model cache, most explicit first:
TATERS_MODEL_CACHEin the environment (an administrator's or a test's word, and the one thing that beats a saved setting);model_cacheinsettings.json(the user's choice in Settings);HF_HUB_CACHEorHF_HOMEfrom the environment (the hub library's own conventions, honored as they always were);- the hub library's default,
~/.cache/huggingface/hub.
Whichever wins is exported as HF_HUB_CACHE at import (see
:func:apply_model_cache), so the hub library, transformers,
sentence-transformers and faster-whisper all download to and read from the
same place -- and it is passed explicitly to the transformer steps too, in
case those libraries were imported before Taters was.
apply_model_cache ¶
apply_model_cache()
Export the chosen cache as HF_HUB_CACHE so every downloading library
agrees with it. Called when Taters is imported. Returns the path when a
Taters-level choice (environment or setting) was applied, else None --
the hub library's own environment is left exactly as found.
Source code in src\taters\helpers\settings.py
164 165 166 167 168 169 170 171 172 173 174 175 176 177 | |
clear_setting ¶
clear_setting(key)
Forget one setting; nothing happens when it was not set.
Source code in src\taters\helpers\settings.py
116 117 118 119 120 121 122 123 124 125 126 127 | |
describe_model_cache ¶
describe_model_cache()
One line for the setup check: the folder, how it was chosen, and how much is in it.
Source code in src\taters\helpers\settings.py
180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 | |
inspect_row_limit ¶
inspect_row_limit()
How many rows to read when looking a spreadsheet over. 0 means all.
All of them by default, because the questions the wizard builds from a spreadsheet are only as true as what it read: on a real file, ten columns looked constant within a group across the first two hundred rows and were not across the other seven hundred, so they were offered as controls that would have come out empty. Reading everything also says at the moment the file is chosen whether its rows match its header, which is much cheaper to learn then than after an hour of extraction.
Lowered by somebody whose files are big enough that a full pass is worth skipping, in Settings. The environment variable wins, for scripts and tests.
Source code in src\taters\helpers\settings.py
53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | |
load_settings ¶
load_settings()
The saved settings, or {}; a damaged file reads as empty rather
than stopping Taters from starting.
Source code in src\taters\helpers\settings.py
92 93 94 95 96 97 98 99 100 | |
model_cache_dir ¶
model_cache_dir()
Where downloaded models live; see the module docstring for the order.
Source code in src\taters\helpers\settings.py
159 160 161 | |
model_cache_source ¶
model_cache_source()
The model cache and which rule chose it: "environment"
(TATERS_MODEL_CACHE), "setting" (chosen in Settings),
"hub environment" (HF_HUB_CACHE/HF_HOME) or "default".
Source code in src\taters\helpers\settings.py
142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | |
save_setting ¶
save_setting(key, value)
Write one setting, keeping the others, atomically.
Source code in src\taters\helpers\settings.py
103 104 105 106 107 108 109 110 111 112 113 | |
settings_path ¶
settings_path()
$TATERS_HOME/settings.json (or ~/.taters/settings.json).
Source code in src\taters\helpers\settings.py
87 88 89 | |
taters.helpers.update_check ¶
"A newer version is out" -- said once, quietly, and never in the way.
The rules this is built around, in order of importance:
- The menu never waits for the network. The note is read from a cached answer, which is instant; the refresh that produces that answer runs in a background thread and its result is for next launch. A slow network, a proxy that blackholes the request, no network at all -- none of it can delay the first screen or stop Taters starting.
- It can be turned off, and turning it off is honored everywhere. Some
people run this on data that cannot leave the building, and an unexplained
outbound connection is a conversation with IT nobody wants. Setting
TATERS_NO_UPDATE_CHECK=1, or the saved setting, stops it dead -- no thread, no request. - It cannot break anything. Every path here is wrapped. A failure leaves no note and no complaint; there is nothing here worth interrupting a run for.
- It says one thing. "Newer version available: v0.7.3" under the banner, in dim text. Not an exhortation, not something to dismiss.
note ¶
note()
The line to print under the banner, or "" -- read from cache, instantly.
Returns "" when: the check is off, Taters is running from a source tree with no version metadata, nothing has been cached yet (the first launch), the cached answer is not newer, or anything at all goes wrong.
Source code in src\taters\helpers\update_check.py
82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | |
refresh_in_background ¶
refresh_in_background()
Start the refresh, or don't. Returns the thread for tests to join.
A daemon thread: if somebody quits Taters two seconds after opening it, the interpreter exits without waiting on a socket nobody is reading.
Source code in src\taters\helpers\update_check.py
139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | |
taters.helpers.feature_columns ¶
What every feature table calls its columns, declared once and checked at build time.
The problem this solves only shows up when two measures meet. Feature tables get
joined side by side into one analysis table, and if two of them use the same
column name, something has to give. stats.assemble already handles that: it
refuses two tables with the same stem, and renames any column used by two tables
to <stem>__<column> in every table that has it, so the outcome does not
depend on file order.
That is a good safety net and it stays. But it is reactive, and reactive
renaming has a property that is poison for research: the name depends on what
else ran. A column is Topic_1 in a pipeline with one topic model and
lda_topics__Topic_1 in a pipeline with two. The same instrument, the same
corpus, two different column names -- so a script written against one study
silently fails against the next, and two results tables cannot be compared
without knowing what else was in each run.
So the names we choose have to be disjoint up front. Each feature module
declares what it writes; :func:overlaps finds any two declarations that could
produce the same name; and tests/test_feature_columns.py fails the build if
any do, naming both offenders. Collisions among shipped measures therefore never
happen, the reactive rename never fires for them, and Topic_1 is Topic_1
in every pipeline forever.
Three kinds of column name exist, and only two of them can be policed here:
- Fixed -- names we choose and always write (
flesch_reading_ease). - Patterned -- names we choose whose tail is a number or a setting
(
Theme_{n},msttr_{n}). Declared as a pattern, matched as one. - Dynamic -- names that come out of the user's own data: the categories in
their dictionary, their archetype names, however many dimensions their encoder
has. We cannot know these and do not pretend to. They are declared with a
sentence saying where they come from, and they are exactly what the reactive
rename in
assembleis for.
Adding a measure means adding a FEATURE_COLUMNS to its module. The test finds
it through the recipe catalog rather than through a list kept here, so forgetting
is a failure rather than a silence.
ColumnSpec
dataclass
¶
ColumnSpec(
label,
names=(),
patterns=(),
reduces_to="Component",
dynamic="",
_module="",
)
One module's claim about the columns it writes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
label
|
str
|
What to call this measure in a collision message. A person reads it. |
required |
names
|
tuple of str
|
Column names always written, spelled exactly. |
()
|
patterns
|
tuple of str
|
Names whose tail varies with a setting, with a placeholder where the
varying part goes. The distinction earns its keep: sentence embeddings write |
()
|
reduces_to
|
str
|
What a PCA over these columns produces, as a singular noun. Reducing a
topic model's topics does not give you "components", it gives you
something worth naming -- so the topic models say |
'Component'
|
dynamic
|
str
|
Non-empty when this module also writes columns whose names come from
the user's data rather than from us. The text says where they come from.
It is documentation, not an exemption: whatever is declared in |
''
|
Notes
Bookkeeping columns are left out on purpose. Several modules write
token_count beside their measures, and declaring it would report a
collision on a column no analysis ever treats as a feature -- assemble
keeps those aside from the feature sets and renames them harmlessly if two
tables carry one. Declare what somebody would analyze.
pattern_regex ¶
pattern_regex(pattern)
A pattern as a regex: everything literal but the one placeholder.
{n} becomes \d+ and {*} becomes .+. Use {n} whenever
the varying part really is a number, because it is what tells e{n} and
e_{n} apart -- two real patterns in this codebase that would otherwise
read as the same one.
Source code in src\taters\helpers\feature_columns.py
129 130 131 132 133 134 135 136 137 138 139 | |
overlaps ¶
overlaps(specs)
Every pair of declarations that could write the same column name.
Returns:
| Type | Description |
|---|---|
list of (label, label, names)
|
One entry per colliding pair, with the names they collide on. Empty when the declarations are disjoint, which is the only acceptable state for the measures Taters ships. |
Source code in src\taters\helpers\feature_columns.py
175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 | |
by_module ¶
by_module(specs)
Declarations keyed by the module that made them, for error messages.
Source code in src\taters\helpers\feature_columns.py
195 196 197 | |
registry ¶
registry()
Every declaration, keyed by module. Built once, and never raises.
A module that cannot be imported (an optional dependency is missing) is skipped rather than fatal: the registry is used to name things nicely, and a missing encoder should not stop a run that never wanted one.
Source code in src\taters\helpers\feature_columns.py
244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 | |
reduced_name ¶
reduced_name(columns, *, set_name='')
What to call a component built from these columns: "Supertopic", or "Component" when nothing more specific is known.
Matched on the column names themselves rather than threaded down from the pipeline, because by the time the statistics stage reduces a feature set, the set is just a list of column names in a table -- whatever produced it is long out of scope.
Source code in src\taters\helpers\feature_columns.py
264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 | |