ArrowStreamDecoderget_next_required_size

This method is provided for users who want to optimize performance. Normal users don’t need to use this method.

Here is an example usage for normal users:

garrow_stream_decoder_consume_buffer(decoder, buffer1);
garrow_stream_decoder_consume_buffer(decoder, buffer2);
garrow_stream_decoder_consume_buffer(decoder, buffer3);

Decoder has internal buffer. If consumed data isn’t enough to advance the state of the decoder, consumed data is buffered to the internal buffer. It causes performance overhead.

If you pass garrow_stream_decoer_get_next_required_size() size data to each garrow_stream_decoder_consume_bytes()/garrow_stream_decoder_consume_buffer() call, the decoder doesn’t use its internal buffer. It improves performance.

Here is an example usage to avoid using internal buffer:

buffer1 = get_data(garrow_stream_decoder_get_next_required_size(decoder));
garrow_stream_decoder_consume_buffer(buffer1);
buffer2 = get_data(garrow_stream_decoder_get_next_required_size(decoder));
garrow_stream_decoder_consume_buffer(buffer2);

Users can use this method to avoid creating small chunks. Record batch data must be contiguous data. If users pass small chunks to the decoder, the decoder needs concatenate small chunks internally. It causes performance overhead.

Here is an example usage to reduce small chunks:

GArrowResizablBuffer *buffer = garrow_resizable_buffer_new(1024, NULL);
while ((small_chunk = get_data(&small_chunk_size))) {
  size_t current_buffer_size = garrow_buffer_get_size(GARROW_BUFFER(buffer));
  garrow_resizable_buffer_resize(buffer, current_buffer_size + small_chunk_size,

NULL); garrow_mutable_buffer_set_data(GARROW_MUTABLE_BUFFER(buffer), current_buffer_size, small_chunk, small_chunk_size, NULL); if (garrow_buffer_get_size(GARROW_BUFFER(buffer)) < garrow_stream_decoder_get_next_required_size(decoder)) { continue; } garrow_stream_decoder_consume_buffer(decoder, GARROW_BUFFER(buffer), NULL); g_object_unref(buffer); buffer = garrow_resizable_buffer_new(1024, NULL); } if (garrow_buffer_get_size(GARROW_BUFFER(buffer)) > 0) { garrow_stream_decoder_consume_buffer(decoder, GARROW_BUFFER(buffer), NULL); } g_object_unref(buffer);

Available since: 18.0.0