Utilities & Helpers¶
Between raw data and analysis sits a layer of unglamorous decisions that quietly shape every number you will eventually publish: what counts as a document, how scattered outputs become one dataset, and how you will prove — months later — where each row came from. The helpers exist so those decisions are made once, explicitly, instead of being improvised in a different way for every project.
If one idea organizes this page, it is the unit of analysis. The same interview corpus can be analyzed per utterance, per speaker, per session, or per person-across-sessions, and the choice changes what your features mean and what your statistics are entitled to claim. Taters makes that choice a visible, repeatable setting rather than an accident of how files happened to be arranged on disk.
- Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10), 1–23. (The "one row per observation" philosophy every helper here serves.)
Finding your files¶
Every project starts with "which files are we even talking about?" — and ad hoc answers (whatever was in the folder that day) are a quiet threat to reproducibility. The file-discovery helper turns that question into an explicit, repeatable specification: what types, which folders, what to include and exclude, and (for media) whether a file actually contains a playable stream rather than merely wearing the right extension.
In Taters: find_files, used internally by every pipeline's input
discovery and available on its own.
From raw text to analysis-ready data¶
Every text method in Taters reads the same simple shape: one row per
document, text_id and text. The gathering helpers produce that shape
from the two ways text actually arrives — a folder of files (one document
each, in any mix of .txt, Word, and PDF), or a spreadsheet with text in one
or more columns.
The spreadsheet path is where the unit-of-analysis decision lives: you can keep one row per original row, or group — say, everything each speaker said across a session becomes one document — and the gatherer records how many pieces went into each group, and can carry along the identifier columns you will need for merging later. It streams, so a spreadsheet far larger than memory is fine, and broken files in a folder (a PDF with no machine-readable text, a mislabeled binary) are skipped with a warning rather than taking the run down.
In Taters: the gather step at the front of every text pipeline; also
callable directly as text_gather.
Merging, stacking and reshaping spreadsheets¶
A lot of data arrives as several files that have to become one before anything can be measured: the trials in one spreadsheet and the participants in another, one file per session, a wide file that should have been long. Until now that meant leaving Taters for R or pandas. Wrangle data now does the four things that cover nearly all of it, and every one of them reads the files first, says in plain words what is going to happen -- with the real numbers -- and only then writes.
Nothing here holds a file in memory. The stacks and the wide-to-long pass stream; the merge and the pivot work through a temporary database on disk, about the size of the file being matched against, which is removed afterward. And nothing here reads a key as a number: a 17-digit ID is never rounded into its neighbor.
Matching two spreadsheets by an ID (merge)¶
Start with the file whose rows you want to keep -- the trials -- then the one with the columns to add -- one row per participant. Tick the column(s) that identify the same thing in both (Taters suggests the one that is in both files and unique in the second). Then it reads both files and tells you what matched:
12,340 of 12,500 rows in trials.csv match a row in participants.csv; 160 do not (e.g. p0091, p1203, p1377).
40 of 540 rows in participants.csv have no match in trials.csv (e.g. p0002, p0310).
age is in both files and agrees on every matched row, so it's kept once.
That read-back is the whole point. A key typed as P001 on one side and 1
on the other is found here, before an analysis is built on it, and when
nothing matches at all the two files' keys are shown side by side so you can
see why.
A few rules, each decided for a reason:
- Keys are compared with surrounding spaces ignored, and nothing else.
When
007on one side is7on the other, orP01isp01, Taters says so and asks whether to match those too -- and says how many rows it would add. It is a question rather than the default because loose matching can also fold two genuinely different IDs together, and the read-back warns when a file has IDs that differ only in case. - One participant, many trials is fine. The merge says each participant row will repeat once per trial and how many rows come out. A key that repeats in both files is refused, with examples, because that almost always means one more column is needed to tell rows apart -- a session, a time point.
- A blank ID matches nothing. It is counted separately from "unmatched".
- A column in both files is kept once when the two agree on every
matched row (a blank on either side counts as agreement, and is filled in
from the other file); otherwise both are kept, the second as
<file>__<column>, and the read-back gives an example of the disagreement. - Which rows come out is asked with real numbers: all 12,500 trials (the 160 without a match get empty participant columns), only the 12,340 that matched, or everything including the 40 participants with no trials. The question is skipped when every row matched.
- IDs that look like
1.23457E+11are what Excel writes after rounding a long number. They are flagged, not matched; re-export that column as text.
Putting spreadsheets one under another (stack)¶
Tick the files. Columns with the same name line up, a column only some files
have is left empty in the others, and a source column says which file each
row came from. Column names that differ only in capitals or spacing (Age,
age) are pointed out and merged on a yes. A file that shares no column with
the rest is pointed out too -- stacked, it is a checkerboard of blanks, and
nearly always what was wanted was a merge.
Wide and long¶
score_t1, score_t2, score_t3 become one score column plus a column saying
which time; or the other way round. Going long, the shared start of the
column names (score_t) is offered for dropping so the time column reads
1, 2, 3, and a row is kept for an empty cell unless you say otherwise, so a
balanced design stays visibly balanced. Going wide, Taters reads the file and
tells you how many rows and new columns will come out; a participant with
two rows for the same condition is shown and you choose whether to average
them or keep the first or last. A column with thousands of values is refused
as the thing to make columns from, because that is nearly always an ID or a
timestamp picked by mistake.
Messy files¶
The files that need this are the messy ones, so the readers expect it. The encoding is worked out from the file -- UTF-8 with or without a BOM, UTF-16 from Excel's "Unicode Text", then Windows cp1252 -- and the separator is sniffed per file, so a semicolon file from a European Excel merges with a comma file without being told. Duplicate or blank column names are made unique and announced. Rows with too few or too many cells are padded or trimmed, counted, and shown before you decide to go on. Writing the result over one of the files being read is refused outright. Every output is UTF-8 and comma-separated, which every other Taters step reads.
In Taters: Wrangle data in the app; also callable directly as
merge_spreadsheets, stack_spreadsheets, wide_to_long and
long_to_wide, each with a plan_* twin that reads the files and reports
without writing anything.
Assembling the final dataset¶
A pipeline leaves you with several feature tables — dictionary scores here, readability there, embeddings somewhere else — that all describe the same documents. The feature gatherer is the last mile: it joins or stacks those tables into the one wide CSV your statistics package expects, and can aggregate on the way (mean embeddings per speaker, say), which is again the unit-of-analysis decision in another costume. Averaging row-level features to person-level features is a modeling choice with real consequences — make it deliberately, and the gatherer makes it visible in the output (group counts ride along).
In Taters: the merge steps a pipeline adds automatically when your
features need aligning; also callable directly as feature_gather.
Practical habits worth stealing¶
- Keep provenance columns. Carrying
source_pathand identifier columns through the gather costs nothing and answers "where did this row come from?" a year later. - Let outputs land where they land. Every tool writes to a predictable
place (
./features/<kind>/...) and refuses to overwrite unless asked — so an interrupted run resumes instead of restarting, and nothing silently replaces the results a draft was written from. - Decide the unit of analysis once, on purpose, and let the pipeline's shared settings enforce it everywhere — rather than aggregating one way in the gather and another way in the merge.