Benchmarking R2R

In the rapidly evolving space of Retrieval-Augmented Generation (RAG), developers face a critical decision: which tool will best serve their project's needs? As more options become available, one factor stands paramount—scalability. This article dives deep into a comparative analysis of R2R against four leading alternatives: LlamaIndex, Haystack, Langchain, and RagFlow, with a focus on performance and throughput.

Our goal is to provide developers with concrete data to inform their choice of RAG tool. By rigorously testing each system's ingestion capabilities, we hope to shed light on their potential to handle real-world, data-intensive applications.

Figure 1: Tokens Ingested vs. Time#

The steeper the better, as the rate indicates faster ingestion

We employed a two-pronged approach to stress-test these systems:

• Bulk Ingestion: We pushed the limits by firehosing approximately ten million tokens of data in batches of 256 files, simulating a high-volume data environment.
• Individual File Processing: To mimic everyday use cases, we measured the ingestion times for single txt and PDF documents, both individually and in multi-file batches. This provides insight into the user experience when adding files to the system.

The results of our benchmarking reveal significant differences in performance across the tested solutions.

Scalable Ingestion#

Our scalable ingestion test aimed to simulate a high-volume data environment, pushing each solution to its limits. This test provides crucial insights into how these tools might perform in data-intensive, real-world scenarios.

We leveraged the HuggingFace Legacy Datasets' Wikipedia corpus for this benchmark. This dataset offers a diverse range of articles, making it an ideal candidate for testing RAG systems. Each framework was tasked with ingesting articles as per their recommended practices, allowing us to measure their performance under optimal conditions.

Note: While R2R supports asynchronous ingestion out of the box, LlamaIndex requires additional configuration for this feature. To ensure a fair comparison, we included results for both the default and asynchronous configurations of LlamaIndex.

The results, summarized in both Table 1 and Figure 1, reveal significant variations in ingestion rates across the tested frameworks. R2R demonstrated the highest throughput, processing over 160,000 tokens per second, while other frameworks showed varying levels of performance.

Table 1: Ingestion Time over 10008026 Tokens

Individual File Ingestion#

To complement our bulk ingestion test, we also examined how each solution handles individual file processing. This test simulates typical user interactions, such as uploading single documents or small batches of files.

Our test suite included two text files of similar size but different content, and two PDF files of identical size but varying complexity. We measured three key metrics for each file:

• Ingestion Time
• Tokens processed per second (for text files)
• Megabytes processed per second

Additionally, we tested combined ingestion of identical file types to assess each solution's performance with multi-file uploads.

Table 2: Time taken to Ingest Files

Table 3: Megabytes per Second Throughput

We encourage developers to conduct their own tests using our publicly available benchmarking code.