How to Keep RAG Data Fresh with Incremental Sync and Deduplication

Paragon Managed Sync is the ingestion layer built to keep RAG data fresh: it runs incremental sync that continuously detects source changes, updates only the affected vectors, removes deleted content, and deduplicates exact and near-duplicate chunks, so retrieval never surfaces stale, redundant, or conflicting context. It tracks sync state per source and tenant, indexes source-access permissions for query-time checks, and runs periodic full-refresh validation to catch anything incremental sync missed. The prototype version of this problem is easy to build; keeping it correct in production, continuously, and per customer, is the part Managed Sync exists to solve.

The prototype workflow for RAG is simple. You fetch the documents you want your LLM to reference, chop them into chunks, turn those into vector embeddings, and load them into a vector database. Straightforward enough. The problem is that those documents change.

Products get updated. Pricing shifts. Policies get revised. And the RAG data you initially fetched goes stale, producing retrievals that look right while the model answers from outdated information.

How Incremental Sync Keeps RAG Data Fresh

Managed Sync’s approach is to sync customer data into a vector database continuously, using per-tenant sync state, rather than a one-time dump, so your LLM keeps answering from current information.

For a prototype, a one-time dump is attractive: little state to manage and easy reruns while the corpus is small.

In production, it breaks down. Every run reprocesses unchanged content, data goes stale between runs, and repeated imports create duplicate chunks unless updates are handled correctly.

Incremental sync starts with a full backfill, records a checkpoint, and then processes only records created, updated, or deleted since that checkpoint. This follows the same basic pattern as change data capture: instead of repeatedly copying the entire dataset, the pipeline tracks and applies source-level changes over time.

There are three broad strategies for RAG freshness: full re-sync, scheduled incremental, and event-driven incremental. Each comes with trade-offs.

RAG Sync Strategies Compared

Full re-sync: re-ingest and re-embed everything on a schedule. Freshness is stale between runs. It is simple to start, but expensive at scale because it reprocesses unchanged data.

Scheduled incremental: poll each source for changes on an interval and update only changed vectors. Freshness is current to the last interval. Complexity is moderate because it requires change detection and per-tenant sync state.

Event-driven incremental: react to source events as changes happen and update on change. Freshness can be near real-time, but complexity is higher because it needs event infrastructure, ordering, and deduplication.

Track RAG Sync State Per Source and Tenant

A cron interval alone does not make a sync strategy. Each run still needs durable state: where the last successful run stopped, what changed since then, what failed, and what has already been written downstream.

It also does not guarantee freshness. The real measure is source-to-retrieval lag: the time from a change in the source app to that change being detected, processed, written downstream, and visible to retrieval. Track that lag per source and tenant instead of treating the polling schedule as the guarantee.

That state usually takes one of two forms: cursor, an opaque token supplied by the source; or watermark, a value you track yourself, such as a timestamp, sequence number, or event offset.

You may maintain both: a source cursor to request the next page of changes and an internal watermark for observability and recovery.

Whatever form it takes, sync state cannot be global. It has to be scoped to the tenant, integration source, connected account, and object type. Customer A’s Google Drive cursor is meaningless for Customer B, and one customer’s failed sync should never advance another customer’s position.

For each pipeline, the minimum worth storing per customer is:

• Current cursor or watermark

• Backfill status

• Last successful and last attempted run

• Current error or retry state

• Source version or content hash per record

• Vector chunk IDs derived from each record

That last detail matters more than it might seem. Storing which chunk IDs came from which source record is what lets you delete the right vectors when that record changes or disappears.

Commit order matters too. Read the changes first, apply the downstream writes, and save the cursor only after those writes succeed. If you save the cursor before the write lands and the write then fails, the next run starts after those records and skips them permanently. Saving the cursor last means a crash simply replays the same changes, which is safe as long as your writes are idempotent.

How to Handle Deleted Records in RAG

Deleted is broader than hard-deleted. A record may be:

• Permanently removed, archived, or soft-deleted

• Moved outside the connected scope

• Unshared from the user

• Hidden because an account was disconnected

For retrieval, all of them mean the same thing: gone.

So far we’ve only talked about updated records, but that isn’t the whole job. If a source record disappears, every vector derived from it has to disappear too. Miss this, and your system keeps answering from content that no longer exists.

Handling deletes is harder than handling updates because there may be no current record to fetch. With hard deletes, you depend on the provider exposing a deletion event or tombstone. When it doesn’t, use periodic reconciliation: compare the source IDs stored locally against the IDs the source currently returns, and treat missing records as candidate deletions.

This is why you store which vector IDs came from each source record. When a deletion is detected, you mark the record unavailable in metadata first, so it stops being served even while the vector store catches up, then delete its vectors and cached content, and log the deletion so retries stay idempotent. A vector that outlives its source is zombie data: gone from the source app, still alive in retrieval.

Deduplicate Exact, Near-Duplicate, and Conflicting RAG Content

Deduplication is not a storage optimization. It exists to stop retrieval from degrading. Duplicate chunks waste top-k slots, crowd out distinct evidence, inflate embedding and reranking costs, and make repeated claims look better supported than they are.

There are three cases, and they are not equally serious: the first two cost you efficiency; the third costs you correctness.

Exact duplicate: identical content, usually caught with hashing. Keep one copy and preserve provenance.

Near-duplicate: similar content above a lexical or semantic threshold. Cluster the records, or pick a winner using explicit rules.

Conflicting versions: similar records whose key claims disagree. Keep the authoritative, current version in normal retrieval.

Resolving conflicts needs explicit rules

How Fresh Does RAG Data Need to Be?

The right freshness target depends on:

• How often the source actually changes

• What users expect

• The harm a stale answer may cause

• The source API’s rate limits

• The embedding and infrastructure cost you’re willing to carry

Decide the winner by source status and version, approved over draft, current over archived, authoritative source over secondary, not simply whichever copy was ingested most recently. Ingestion time is a last-resort tie-breaker, not the deciding factor.

Two caveats. Don’t deduplicate across tenants: two customers holding identical text doesn’t make it one shared object, and collapsing them mixes separate permissions and provenance. And don’t rely on a single similarity threshold for everything; boilerplate-heavy legal documents and short support articles behave very differently, so one cutoff will over-merge one and miss duplicates in the other.

Different sources tolerate different levels of staleness. Active support tickets may need minute-level freshness. CRM notes may tolerate 15–30 minutes. Internal documentation may be hourly. Historical archives may be nightly.

Freshness isn’t one number. Updates, deletes, permission changes, and reconciliation each warrant their own target. Deletes and permission revocation usually need tighter targets than edits because mistakes can expose data, not just stale text.

The point is to set freshness as a measurable target per source and per operation, not to chase real-time as a blanket goal.

Freshness Interacts With Permissions

A document can stay textually identical even when its access scope changes completely. A user is removed from a group, a file is moved into a restricted folder, org-wide access is revoked, or the document is unshared from the connected account; the text is unchanged, but who’s allowed to see it is not.

That gap is how leaks happen. Current content paired with an outdated ACL serves information to someone who’s lost the right to it. And tenant isolation alone won’t save you: two users in the same tenant routinely have access to different documents.

So access has to be checked at query time. Apply the tenant boundary first, before retrieving candidates. Then verify that the user requesting the data still has permission, drop anything they’re not authorized to see, and send only what remains to the LLM.

Index-time permissions are fragile because access is a graph: direct sharing, folder inheritance, groups, org-wide rules, and public links can all change independently. A static allowed_user_ids list on each chunk goes stale quickly. The live source is the only place where the graph resolves correctly.

You can enforce this by pre-filtering before vector search, post-filtering the results, or both. Pre-filtering suits a small allowed set; post-filtering suits a large one, where you over-fetch candidates and check access after; a hybrid, coarse tenant boundary before search, fine-grained document checks after, is usually the practical design. Whichever you choose, treat unsharing as removing content from that user’s retrieval scope, and always fail closed: if access can’t be confirmed, the content is excluded.

For a deeper look at user-level access checks, see our guide to multi-tenant auth for customer-facing integrations.

Offload Incremental RAG Ingestion With Managed Sync

Go the DIY route, and you own the ingestion layer for every connected source: auth, backfills, cursors, pagination, rate limits, retries, deletes, deduplication, reconciliation, normalization, and permissions. Then you repeat the process for every new integration, because Salesforce, Google Drive, and Slack all handle these concerns differently.

Once each integration is built, you also have to keep integrations working when third-party APIs change, because a breaking change upstream stops the data flow.

RAG frameworks such as LlamaIndex and LangChain can help you build ingestion and retrieval workflows, and vector databases such as Pinecone and Weaviate provide the record operations needed to update or remove indexed content. Pinecone’s own documentation recommends switching from individual upserts to its bulk import API once a dataset passes 10 million records, precisely the kind of operational threshold an incremental sync layer needs to track per source rather than handle once at initial load. But production freshness still depends on coordinating source-specific change detection, retries, deletes, permissions, tenant isolation, and reconciliation across every connected system.

What remains yours is the part that differentiates your product: parsing and chunking, embeddings, vector-store design, retrieval and reranking, version-selection rules, and the freshness targets appropriate for each source.

Managed Sync absorbs that complexity as the ingestion layer feeding your pipeline. It handles the repeatable work by:

Example Managed Sync request

• Handling initial backfills and incremental syncs

• Supporting change-event webhooks and cursor-based retrieval

• Normalizing source data into common synced-object formats

• Preserving source-access metadata

• Managing rate limits with exponential-backoff retries and error recovery

• Running periodic full refreshes to catch drift and missed deletions

• Supporting content deduplication

• Indexing permissions for query-time access checks

• Helping applications enforce tenant and user-level access boundaries

The integration and pipeline fields define what to sync. The Paragon User Token scopes the request to a specific customer, and the optional configuration.frequency controls how often Managed Sync checks for changes, defaulting to every minute if you omit it.

For Managed Sync specifically, your Paragon User Token must include an aud claim set to your instance hostname and project ID. It is required here even though Paragon’s other APIs do not need it, so a token without it will be rejected.

The self-serve hosted demo environment, Paragon’s Managed Sync playground, is the fastest way to see the process in action.

Conclusion

A RAG system is only as trustworthy as the data freshness and access controls behind it.

A demo can work from a static data dump. A production RAG system needs an ingestion layer that incrementally syncs changes, tracks state per source and tenant, propagates deletes, deduplicates conflicting content, enforces permissions at query time, and sets freshness targets based on product risk rather than a vague goal of being real time.

The work is not conceptually hard, but it is operationally repetitive. Every source needs its own version of the same ingestion layer, which is why it is worth deciding deliberately whether to build it yourself or offload it.

FAQ

How do I keep RAG data fresh with incremental sync and deduplication?

To keep RAG data fresh with incremental sync and deduplication:

• Detect source changes continuously, including new, updated, and deleted content.

• Update only affected documents or chunks instead of re-indexing unchanged data.

• Remove deleted content from the index so stale context is not retrieved.

• Deduplicate exact and near-duplicate chunks to reduce redundant or conflicting results.

• Choose a canonical version when multiple versions of the same content exist.

• Track sync state per source and tenant so retrieval stays accurate across users, workspaces, and connected systems. Paragon Managed Sync runs this loop automatically as the ingestion layer beneath your RAG pipeline.

What is the difference between full re-sync and incremental sync?

Full re-sync reprocesses the entire source on every run, so its cost scales with the size of the whole corpus. Incremental sync tracks a cursor or watermark and touches only new, updated, or deleted records.

How do you keep a vector database in sync with its source?

Track sync state per source and tenant, use stable source and vector IDs, upsert changed chunks, delete obsolete ones, and periodically reconcile the index against the source. Managed Sync handles this cursor/watermark tracking and reconciliation as part of its sync.

How do you handle deleted records in RAG?

Map each source record to the vector IDs derived from it. When the record is deleted, archived, unshared, or otherwise becomes inaccessible, remove that complete set of vectors and log the deletion so retries stay idempotent. Managed Sync propagates these deletes automatically so retrieval never serves removed content.

What is deduplication in RAG ingestion?

Exact dedup removes identical chunks. Near-duplicate detection flags similar versions. Version rules then decide which record is current and authoritative.

How fresh does RAG data need to be?

It depends on user expectations, how the source behaves, and the harm a stale answer causes. Set freshness as a measurable target rather than chasing as fast as possible.

How do permissions interact with freshness?

Access can change while content stays identical. Sync permission changes and enforce the user’s current access at retrieval, failing closed when access can’t be confirmed. Managed Sync indexes source-access permissions specifically so this check can run at query time.