Performance

Method

bench/harness.php runs each workload twice: as a phplyr pipeline, and as the same calls made directly on the table. bench/generate.php writes the data. It is the same generator and seed as phibble’s bin/bench, so the two harnesses time identical data:

• events.csv: 1,000,000 rows of id, key (10,000 distinct values), g (8 labels), x (float, about 2% NA), and t (datetime).
• dims.csv: 100,000 rows of key, label, and w.

php bench/generate.php 1000000
php bench/harness.php --backend=dataframe --runs=3
php bench/harness.php --backend=sqlite --runs=3

Each timing includes collect(), so lazy backends do their work inside the timer. The numbers below are medians of three runs on an Apple Silicon Mac with PHP 8.5 and phibble fa978f6.

phplyr’s overhead

At 100,000 rows, the phplyr pipeline and the direct calls differed by at most 5% on every workload and backend, which is within run-to-run noise. Grouped slices, pivot_*(), and complete() build on the same table methods and carry the same small overhead.

Timings at 1,000,000 rows

DataFrame peaked at about 1.5 GB of memory for the 1M-row table and its results.

Choosing a backend

• In-memory DataFrame suits data that fits in memory, up to a few million rows. Expression arguments are the fastest form: a grouped summary with mean('x') runs about four times faster than the same summary written as closures, because phibble computes it in one pass without building a table per group.

• A database table (Phibble\tbl()) avoids loading the data into PHP. With expressions, filter() and summarise() run in the database, and only the result reaches PHP, so memory stays small. This is the right choice when the data lives in a database or doesn’t fit in memory. For data already in memory, a data frame is faster: these timings exclude loading, and reading the 1M-row CSV into a data frame takes about 1.4 seconds. Pipelines that return every row (mutate(), arrange(), unfiltered joins) spend most of their database time transferring rows to PHP.

• Closures on a database table load all rows first and then run in memory. Use col() expressions and phplyr’s helpers to keep the work in SQL.

• SQLite columns need declared types. PDO binds values as strings. In an untyped column, SQLite stores them as text and compares numbers as text.

• PolarsTable (phibble’s Polars backend, on the phpolars extension) runs the pipeline in Polars, 10 to 100 times faster than DataFrame, and pays mainly for bringing rows back into PHP. See below.

PolarsTable

bench/backends.php times three phplyr pipelines on DataFrame, PolarsTable, and SQLite, and checks that all three backends return the same rows. It is adapted from phpolars’ benchmark script. The pipelines are:

• A: filter, join a store lookup, two-key summarise, then a ranking window.
• B: grouped lag(orderBy:), cumsum(), and a share of the group total over all rows.
• C: pipeline A with a closure in filter().

Times are the best of three runs through collect(), on an Apple M5 Ultra with Homebrew PHP 8.5, phibble 0.3.0, and Polars 0.55.2 at two threads. In every case the three backends returned the same rows. The full results, with all repeat timings and version details, are in benchmarks/backends-results.json.

bench/backends.R runs pipelines A and B with dplyr 1.2 on the same data, and checks that dplyr’s row counts and column sums match phplyr’s. Its results are in benchmarks/dplyr-results.json. Pipeline C has no dplyr version, because its closure is PHP code.

dplyr runs these pipelines in compiled code. On A, which reduces the data before collecting, PolarsTable (0.05 s) and dplyr (0.07 s) take about the same time. On B, which returns every row, dplyr is the fastest by a wide margin.

The operations themselves are fast in Polars. Most of PolarsTable’s time goes to converting results back to PHP values: about 0.2 seconds per million rows of plain columns, plus about 0.5 seconds per million values of a datetime column, because each value becomes a PHP DateTimeImmutable. A pipeline that returns every row of a table with a datetime column can therefore be slower on PolarsTable than on DataFrame. In phibble’s harness, mutate() and left_join() returning 1M rows took 1.3 seconds on PolarsTable against 0.3 seconds on DataFrame.

A closure makes PolarsTable convert the data to a DataFrame and back, which costs about a second per million rows. Pipeline C runs its closure step that way.

Use PolarsTable when a pipeline reduces the data before collecting (summaries, filters, distinct(), or a join followed by a summary), when it chains several steps, or when the data starts in Polars: PolarsTable::fromPolars(Phpolars\read_csv($path)) skips phibble’s CSV parse. Keep the pipeline in expressions so every step stays in Polars.