pyarrow.dataset.ParquetFileFragment

class pyarrow.dataset.ParquetFileFragment

Bases: FileFragment

A Fragment representing a parquet file.

__init__(*args, **kwargs)

Methods

__init__(*args, **kwargs)
count_rows(self, Expression filter=None, ...)	Count rows matching the scanner filter.
ensure_complete_metadata(self)	Ensure that all metadata (statistics, physical schema, ...) have been read and cached in this fragment.
head(self, int num_rows[, columns])	Load the first N rows of the fragment.
open(self)	Open a NativeFile of the buffer or file viewed by this fragment.
scanner(self, Schema schema=None[, columns])	Build a scan operation against the fragment.
split_by_row_group(self, ...)	Split the fragment into multiple fragments.
subset(self, Expression filter=None, ...[, ...])	Create a subset of the fragment (viewing a subset of the row groups).
take(self, indices[, columns])	Select rows of data by index.
to_batches(self, Schema schema=None[, columns])	Read the fragment as materialized record batches.
to_table(self, Schema schema=None[, columns])	Convert this Fragment into a Table.

Attributes

buffer	The buffer viewed by this fragment, if it views a buffer.
filesystem	The FileSystem containing the data file viewed by this fragment, if it views a file.
format	The format of the data file viewed by this fragment.
metadata
num_row_groups	Return the number of row groups viewed by this fragment (not the number of row groups in the origin file).
partition_expression	An Expression which evaluates to true for all data viewed by this Fragment.
path	The path of the data file viewed by this fragment, if it views a file.
physical_schema	Return the physical schema of this Fragment.
row_groups

buffer

The buffer viewed by this fragment, if it views a buffer. If instead it views a file, this will be None.

count_rows(self, Expression filter=None, int batch_size=_DEFAULT_BATCH_SIZE, int batch_readahead=_DEFAULT_BATCH_READAHEAD, int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD, FragmentScanOptions fragment_scan_options=None, bool use_threads=True, MemoryPool memory_pool=None)

Count rows matching the scanner filter.

Parameters:

filterExpression, default None

Scan will return only the rows matching the filter. If possible the predicate will be pushed down to exploit the partition information or internal metadata found in the data source, e.g. Parquet statistics. Otherwise filters the loaded RecordBatches before yielding them.

batch_sizeint, default 131_072

The maximum row count for scanned record batches. If scanned record batches are overflowing memory then this method can be called to reduce their size.

batch_readaheadint, default 16

The number of batches to read ahead in a file. This might not work for all file formats. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_readaheadint, default 4

The number of files to read ahead. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_scan_optionsFragmentScanOptions, default None

Options specific to a particular scan and fragment type, which can change between different scans of the same dataset.

use_threadsbool, default True

If enabled, then maximum parallelism will be used determined by the number of available CPU cores.

memory_poolMemoryPool, default None

For memory allocations, if required. If not specified, uses the default pool.

Returns:

countint

ensure_complete_metadata(self)

Ensure that all metadata (statistics, physical schema, …) have been read and cached in this fragment.

filesystem

The FileSystem containing the data file viewed by this fragment, if it views a file. If instead it views a buffer, this will be None.

format

The format of the data file viewed by this fragment.

head(self, int num_rows, columns=None, Expression filter=None, int batch_size=_DEFAULT_BATCH_SIZE, int batch_readahead=_DEFAULT_BATCH_READAHEAD, int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD, FragmentScanOptions fragment_scan_options=None, bool use_threads=True, MemoryPool memory_pool=None)

Load the first N rows of the fragment.

Parameters:

num_rowsint

The number of rows to load.

columnslist of str, default None

The columns to project. This can be a list of column names to include (order and duplicates will be preserved), or a dictionary with {new_column_name: expression} values for more advanced projections.

The list of columns or expressions may use the special fields __batch_index (the index of the batch within the fragment), __fragment_index (the index of the fragment within the dataset), __last_in_fragment (whether the batch is last in fragment), and __filename (the name of the source file or a description of the source fragment).

The columns will be passed down to Datasets and corresponding data fragments to avoid loading, copying, and deserializing columns that will not be required further down the compute chain. By default all of the available columns are projected. Raises an exception if any of the referenced column names does not exist in the dataset’s Schema.

filterExpression, default None

Scan will return only the rows matching the filter. If possible the predicate will be pushed down to exploit the partition information or internal metadata found in the data source, e.g. Parquet statistics. Otherwise filters the loaded RecordBatches before yielding them.

batch_sizeint, default 131_072

The maximum row count for scanned record batches. If scanned record batches are overflowing memory then this method can be called to reduce their size.

batch_readaheadint, default 16

The number of batches to read ahead in a file. This might not work for all file formats. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_readaheadint, default 4

The number of files to read ahead. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_scan_optionsFragmentScanOptions, default None

Options specific to a particular scan and fragment type, which can change between different scans of the same dataset.

use_threadsbool, default True

If enabled, then maximum parallelism will be used determined by the number of available CPU cores.

memory_poolMemoryPool, default None

For memory allocations, if required. If not specified, uses the default pool.

Returns:

Table

metadata

num_row_groups

Return the number of row groups viewed by this fragment (not the number of row groups in the origin file).

open(self)

Open a NativeFile of the buffer or file viewed by this fragment.

partition_expression

An Expression which evaluates to true for all data viewed by this Fragment.

path

The path of the data file viewed by this fragment, if it views a file. If instead it views a buffer, this will be “<Buffer>”.

physical_schema

Return the physical schema of this Fragment. This schema can be different from the dataset read schema.

row_groups

scanner(self, Schema schema=None, columns=None, Expression filter=None, int batch_size=_DEFAULT_BATCH_SIZE, int batch_readahead=_DEFAULT_BATCH_READAHEAD, int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD, FragmentScanOptions fragment_scan_options=None, bool use_threads=True, MemoryPool memory_pool=None)

Build a scan operation against the fragment.

Data is not loaded immediately. Instead, this produces a Scanner, which exposes further operations (e.g. loading all data as a table, counting rows).

Parameters:

schemaSchema

Schema to use for scanning. This is used to unify a Fragment to its Dataset’s schema. If not specified this will use the Fragment’s physical schema which might differ for each Fragment.

columnslist of str, default None

The columns to project. This can be a list of column names to include (order and duplicates will be preserved), or a dictionary with {new_column_name: expression} values for more advanced projections.

The list of columns or expressions may use the special fields __batch_index (the index of the batch within the fragment), __fragment_index (the index of the fragment within the dataset), __last_in_fragment (whether the batch is last in fragment), and __filename (the name of the source file or a description of the source fragment).

The columns will be passed down to Datasets and corresponding data fragments to avoid loading, copying, and deserializing columns that will not be required further down the compute chain. By default all of the available columns are projected. Raises an exception if any of the referenced column names does not exist in the dataset’s Schema.

filterExpression, default None

Scan will return only the rows matching the filter. If possible the predicate will be pushed down to exploit the partition information or internal metadata found in the data source, e.g. Parquet statistics. Otherwise filters the loaded RecordBatches before yielding them.

batch_sizeint, default 131_072

The maximum row count for scanned record batches. If scanned record batches are overflowing memory then this method can be called to reduce their size.

batch_readaheadint, default 16

The number of batches to read ahead in a file. This might not work for all file formats. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_readaheadint, default 4

The number of files to read ahead. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_scan_optionsFragmentScanOptions, default None

Options specific to a particular scan and fragment type, which can change between different scans of the same dataset.

use_threadsbool, default True

If enabled, then maximum parallelism will be used determined by the number of available CPU cores.

memory_poolMemoryPool, default None

For memory allocations, if required. If not specified, uses the default pool.

Returns:

scannerScanner

split_by_row_group(self, Expression filter=None, Schema schema=None)

Split the fragment into multiple fragments.

Yield a Fragment wrapping each row group in this ParquetFileFragment. Row groups will be excluded whose metadata contradicts the optional filter.

Parameters:

filterExpression, default None

Only include the row groups which satisfy this predicate (using the Parquet RowGroup statistics).

schemaSchema, default None

Schema to use when filtering row groups. Defaults to the Fragment’s physical schema

Returns:

A list of Fragments

subset(self, Expression filter=None, Schema schema=None, row_group_ids=None)

Create a subset of the fragment (viewing a subset of the row groups).

Subset can be specified by either a filter predicate (with optional schema) or by a list of row group IDs. Note that when using a filter, the resulting fragment can be empty (viewing no row groups).

Parameters:

filterExpression, default None

Only include the row groups which satisfy this predicate (using the Parquet RowGroup statistics).

schemaSchema, default None

Schema to use when filtering row groups. Defaults to the Fragment’s physical schema

row_group_idslist of ints

The row group IDs to include in the subset. Can only be specified if filter is None.

Returns:

ParquetFileFragment

take(self, indices, columns=None, Expression filter=None, int batch_size=_DEFAULT_BATCH_SIZE, int batch_readahead=_DEFAULT_BATCH_READAHEAD, int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD, FragmentScanOptions fragment_scan_options=None, bool use_threads=True, MemoryPool memory_pool=None)

Select rows of data by index.

Parameters:

indicesArray or array-like

The indices of row to select in the dataset.

columnslist of str, default None

The columns to project. This can be a list of column names to include (order and duplicates will be preserved), or a dictionary with {new_column_name: expression} values for more advanced projections.

The list of columns or expressions may use the special fields __batch_index (the index of the batch within the fragment), __fragment_index (the index of the fragment within the dataset), __last_in_fragment (whether the batch is last in fragment), and __filename (the name of the source file or a description of the source fragment).

The columns will be passed down to Datasets and corresponding data fragments to avoid loading, copying, and deserializing columns that will not be required further down the compute chain. By default all of the available columns are projected. Raises an exception if any of the referenced column names does not exist in the dataset’s Schema.

filterExpression, default None

Scan will return only the rows matching the filter. If possible the predicate will be pushed down to exploit the partition information or internal metadata found in the data source, e.g. Parquet statistics. Otherwise filters the loaded RecordBatches before yielding them.

batch_sizeint, default 131_072

The maximum row count for scanned record batches. If scanned record batches are overflowing memory then this method can be called to reduce their size.

batch_readaheadint, default 16

The number of batches to read ahead in a file. This might not work for all file formats. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_readaheadint, default 4

The number of files to read ahead. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_scan_optionsFragmentScanOptions, default None

Options specific to a particular scan and fragment type, which can change between different scans of the same dataset.

use_threadsbool, default True

If enabled, then maximum parallelism will be used determined by the number of available CPU cores.

memory_poolMemoryPool, default None

For memory allocations, if required. If not specified, uses the default pool.

Returns:

Table

to_batches(self, Schema schema=None, columns=None, Expression filter=None, int batch_size=_DEFAULT_BATCH_SIZE, int batch_readahead=_DEFAULT_BATCH_READAHEAD, int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD, FragmentScanOptions fragment_scan_options=None, bool use_threads=True, MemoryPool memory_pool=None)

Read the fragment as materialized record batches.

Parameters:

schemaSchema, optional

Concrete schema to use for scanning.

columnslist of str, default None

The columns to project. This can be a list of column names to include (order and duplicates will be preserved), or a dictionary with {new_column_name: expression} values for more advanced projections.

The list of columns or expressions may use the special fields __batch_index (the index of the batch within the fragment), __fragment_index (the index of the fragment within the dataset), __last_in_fragment (whether the batch is last in fragment), and __filename (the name of the source file or a description of the source fragment).

The columns will be passed down to Datasets and corresponding data fragments to avoid loading, copying, and deserializing columns that will not be required further down the compute chain. By default all of the available columns are projected. Raises an exception if any of the referenced column names does not exist in the dataset’s Schema.

filterExpression, default None

Scan will return only the rows matching the filter. If possible the predicate will be pushed down to exploit the partition information or internal metadata found in the data source, e.g. Parquet statistics. Otherwise filters the loaded RecordBatches before yielding them.

batch_sizeint, default 131_072

The maximum row count for scanned record batches. If scanned record batches are overflowing memory then this method can be called to reduce their size.

batch_readaheadint, default 16

The number of batches to read ahead in a file. This might not work for all file formats. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_readaheadint, default 4

The number of files to read ahead. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_scan_optionsFragmentScanOptions, default None

Options specific to a particular scan and fragment type, which can change between different scans of the same dataset.

use_threadsbool, default True

If enabled, then maximum parallelism will be used determined by the number of available CPU cores.

memory_poolMemoryPool, default None

For memory allocations, if required. If not specified, uses the default pool.

Returns:

record_batchesiterator of RecordBatch

to_table(self, Schema schema=None, columns=None, Expression filter=None, int batch_size=_DEFAULT_BATCH_SIZE, int batch_readahead=_DEFAULT_BATCH_READAHEAD, int fragment_readahead=_DEFAULT_FRAGMENT_READAHEAD, FragmentScanOptions fragment_scan_options=None, bool use_threads=True, MemoryPool memory_pool=None)

Convert this Fragment into a Table.

Use this convenience utility with care. This will serially materialize the Scan result in memory before creating the Table.

Parameters:

schemaSchema, optional

Concrete schema to use for scanning.

columnslist of str, default None

The columns to project. This can be a list of column names to include (order and duplicates will be preserved), or a dictionary with {new_column_name: expression} values for more advanced projections.

The list of columns or expressions may use the special fields __batch_index (the index of the batch within the fragment), __fragment_index (the index of the fragment within the dataset), __last_in_fragment (whether the batch is last in fragment), and __filename (the name of the source file or a description of the source fragment).

The columns will be passed down to Datasets and corresponding data fragments to avoid loading, copying, and deserializing columns that will not be required further down the compute chain. By default all of the available columns are projected. Raises an exception if any of the referenced column names does not exist in the dataset’s Schema.

filterExpression, default None

Scan will return only the rows matching the filter. If possible the predicate will be pushed down to exploit the partition information or internal metadata found in the data source, e.g. Parquet statistics. Otherwise filters the loaded RecordBatches before yielding them.

batch_sizeint, default 131_072

The maximum row count for scanned record batches. If scanned record batches are overflowing memory then this method can be called to reduce their size.

batch_readaheadint, default 16

The number of batches to read ahead in a file. This might not work for all file formats. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_readaheadint, default 4

The number of files to read ahead. Increasing this number will increase RAM usage but could also improve IO utilization.

fragment_scan_optionsFragmentScanOptions, default None

Options specific to a particular scan and fragment type, which can change between different scans of the same dataset.

use_threadsbool, default True

If enabled, then maximum parallelism will be used determined by the number of available CPU cores.

memory_poolMemoryPool, default None

For memory allocations, if required. If not specified, uses the default pool.

Returns:

tableTable