Sync customer data into a vector database

Pull, normalize, and permission customer data before it hits your vector database, and where Managed Sync fits versus a dedicated embedding pipeline.

Garrett Scott

,

Head of Marketing

Sync customer data into a vector database

Managed Sync is Paragon's answer to the upstream job every RAG pipeline built on customer data has to solve first: it backfills and normalizes sources across categories such as file storage, documents, and CRM into one Synced Object schema, and its Permissions API lets your app check a user's access before it serves a record, so what reaches your pipeline is already clean and permissioned. What it does not do is embed that data or write it into a vector database. Choosing an embedding model and a destination like Pinecone or Weaviate is your pipeline to build; Managed Sync is, in Paragon's own docs, "the beginning of your RAG pipeline." Before a single embedding gets generated, every RAG pipeline built on customer data hits the same wall: the data has to be pulled out of wherever a customer's team keeps it (a shared drive, a wiki, a CRM), normalized into something consistent, kept current as it changes, and scoped so one user's retrieval never surfaces a document another user can't see. Building that per source, for every customer, before you've written a line of embedding code, is a bigger job than most teams budget for, and it's the job Managed Sync exists to solve.

What makes syncing customer data into a vector database hard

Paragon's Managed Sync exists because this is a pipeline problem, not an API call, and it stays that way for as long as the RAG system is live. Four things make it hard in practice, and they compound with every new source added from Paragon's connector catalog. Every source authenticates differently, and a customer's user has to grant access per source rather than once. Every source's schema is different: a CRM's contact record, a Drive folder, and a Notion page share almost no fields in common, so "sync everything" means writing a normalizer for each one. Access control has to survive all the way into retrieval, not just into the sync job, or a permission that exists in the source system quietly disappears once the data sits in your index. And the data goes stale the moment it's pulled unless something is watching for changes and re-pulling them, which turns a one-time migration into a standing piece of infrastructure. Managed Sync is built to own all four: one Synced Object schema per category instead of a normalizer per source, one Permissions API instead of a permission check your team writes and maintains per provider, and backfill plus near-real-time incremental sync instead of a batch job that goes stale between runs.

What Managed Sync does, and does not do

Managed Sync pulls a customer's data from a connected source, normalizes it into a Synced Object, and gates it with the Permissions API before your app ever reads it. It does not generate embeddings, and it does not write to a vector database. There is no Pinecone destination, no Weaviate destination, and no embedding step anywhere in the product. Once a record clears the Permissions API and comes back through the Sync API, what happens next, meaning chunking, embedding, and storage in whatever vector database you run, is your pipeline, on your models and your infrastructure. Paragon's own documentation states the scope directly: Managed Sync's Sync API "can be used as the beginning of your RAG pipeline," not as a replacement for the rest of it.

Embedding models and vector database choices are decisions tied to your product's retrieval quality and cost, and they change faster than the ingestion layer underneath them. Keeping the two separate means switching from one embedding model to another, or moving a vector database from one vendor to another, doesn't require re-authorizing every customer's data source or rebuilding the permission logic that gates access to it. It also means Managed Sync isn't tied to whichever vector database vendors it happens to have built a destination for. If a platform's whole pitch is a native vector-database destination, that's the tradeoff running the other way: convenient if the destination it supports is the one you want, a constraint if it isn't.

Permission-aware retrieval, before anything gets embedded

The reason this matters before embedding, not after, is that once a document is embedded and sitting in a vector store, the permission check on it is only as good as whatever your retrieval code enforces at query time. Paragon's Permissions API builds an access graph from each source's own permission model, so a record that comes back through the Sync API already carries the access rule it had in Drive, Notion, or the CRM it came from. Your retrieval layer checks that specific user's access before serving a document or chunk, rather than trusting that everyone with query access to the index is allowed to see everything in it. The deep mechanics of that check, including how the access graph gets built and what a permission-aware query looks like, are covered in building AI agents that integrate with a customer's external data; the short version here is that the permission travels with the record, so it's still enforceable at retrieval time no matter what vector database ends up holding the embedding.

Keeping the source data current

A vector database is only as useful as the data it was built from is current, and Managed Sync's job is making sure that source data doesn't go stale underneath your embeddings. It backfills a customer's historical data from any connected source with a single API call, so there's something to embed from day one rather than only what accumulates after the connection is made. From there, near-real-time incremental updates catch new and changed records without a full re-sync, and intelligent retry and backoff handle the rate limits every third-party API eventually throws back. A periodic full-refresh validation catches anything incremental sync might have missed. None of that freshness work touches your vector database directly. It keeps the upstream Synced Object current, so whenever your pipeline re-embeds or re-indexes, it's working from data that reflects the source, not a snapshot from whenever the last sync happened to run.

How Paragon fits into your RAG pipeline

Here's the same flow end to end, with the handoff point made explicit. Paragon owns the middle; your pipeline owns the last step.

Connect. A customer's user authorizes a data source, such as a shared drive, a wiki, or a CRM, through Paragon's managed authentication (see multi-tenant auth for customer-facing integrations for how the token model isolates one customer's credentials from another's).
Backfill and sync. Managed Sync backfills that user's historical records with a single API call, then keeps them current with near-real-time incremental updates as the source changes.
Normalize. Records come back as a Synced Object, one schema per category, among them file storage, documents, and CRM (including Salesforce and HubSpot data synced this way), instead of a different shape per provider.
Gate access. The Permissions API checks that specific user's access before a record is served, so what your pipeline receives is already scoped to what that person is allowed to see.
Embed and store. Your pipeline takes it from there: chunking, embedding with whatever model you choose, and upserting into whatever vector database you run.

Failure handling in that flow is mostly about what happens when a source connection breaks. If a user's authorization lapses, that surfaces back through the Connect Portal instead of silently returning stale or partial data, and Managed Sync's retry and backoff logic absorbs upstream rate-limit errors rather than dropping records on the floor. Your pipeline sees a clean, permissioned Synced Object either way, or an explicit signal that a source needs reauthorization, not a silent gap in what got indexed.

Where Paragon fits versus a dedicated embedding pipeline

For the ingestion and permissions layer feeding a vector database, meaning the part that pulls, normalizes, and permission-gates a customer's data before anything gets embedded, Managed Sync is the pick. For the embed-and-upsert step itself, that's your own pipeline's job, whether you write that code yourself or use a tool built for it, such as Airbyte, an open-source ELT platform with 600-plus connectors current as of July 2026. Its Pinecone destination connector pulls data from a connected source, embeds it using OpenAI's text-embedding-ada-002 or Cohere's embed-english-light-v2.0, and upserts the result directly into the index. That's a genuine embed-and-upsert pipeline, which is a different piece of the stack than what Managed Sync does.

	Paragon Managed Sync	Airbyte
What's normalized upstream	Sources across categories such as file storage, documents, and CRM into one Synced Object schema	Source data extracted through its connector catalog
Permission-aware retrieval	Permissions API builds an access graph from each source's own permission model	Not a feature of the platform; access control is left to whatever reads the destination afterward
Embeddings generated	None. Managed Sync does not generate embeddings	Yes, through its Pinecone and Weaviate destination connectors (OpenAI or Cohere models)
Vector database destination	None named. Output is a normalized, permissioned record your pipeline embeds and writes	Native destination connectors that embed and upsert directly into Pinecone or Weaviate
Freshness mechanics	Automatic backfill, near-real-time incremental updates, retry and backoff on rate limits	Sync schedules configured per connection
Best fit	The ingestion and permissions layer upstream of embedding	The embed-and-upsert step, once source data is already flowing through a supported connection

Many teams end up using both pieces of this stack: Managed Sync for the auth, normalization, and permission work against dozens of live, permissioned SaaS sources, and a separate embedding step, either their own code or a tool like Airbyte, for the part that turns clean records into vectors in a specific database.

FAQ

Does Paragon write data into a vector database? No. Paragon's Managed Sync pulls, normalizes, and permission-gates a customer's data and hands it back through the Sync API. It does not generate embeddings and does not write to Pinecone, Weaviate, or any other vector database. That step is your pipeline, built on your choice of embedding model and destination.

What does Managed Sync actually normalize? Managed Sync normalizes sources across categories such as file storage, documents, and CRM, each into its own Synced Object schema, so a record from a shared drive, a wiki, or a CRM comes back in a consistent shape regardless of which specific provider it came from. That's what lets your pipeline write one embedding process instead of a normalizer per source.

How does permission-aware retrieval work before anything gets embedded? Paragon's Permissions API builds an access graph from each source's own RBAC model, so a record carries its access rule into the Sync API response. Your app checks that specific user's access before serving a document or chunk, instead of trusting that anyone who can query the index is allowed to see everything in it.

How fresh is the data my vector database gets built from? Managed Sync backfills historical data from any connected source with a single API call, then keeps it current with near-real-time incremental updates and retry-and-backoff handling for upstream rate limits, with periodic full-refresh validation as a backstop. That keeps the upstream Synced Object current between whatever re-embedding schedule your pipeline runs.

What's the difference between Paragon Managed Sync and Airbyte's Pinecone connector, current as of July 2026? Airbyte's Pinecone and Weaviate destination connectors embed source data (using OpenAI or Cohere models) and upsert it directly, an embed-and-upsert pipeline Managed Sync does not offer. Managed Sync's job is upstream of that: pulling, normalizing, and permission-gating data from dozens of live SaaS sources before anything is embedded, with a Permissions API that Airbyte's destination connectors don't provide.

Do I still need to build an embedding pipeline if I use Managed Sync? Yes. Managed Sync hands you a clean, permissioned, current record through the Sync API. Chunking, choosing an embedding model, and writing to a vector database are steps you or your pipeline still owns. Paragon is SOC 2 Type II, GDPR, and HIPAA compliant, which covers the ingestion and permissions layer; the embedding step downstream is outside that layer's scope.

In short

Getting a customer's data into a vector database for RAG starts with pulling that data reliably, permission-gated, and current from every source it lives in. Managed Sync owns that part: normalizing sources across categories such as file storage, documents, and CRM into one Synced Object schema, gating access with the Permissions API, and keeping data current with backfill plus near-real-time incremental sync. Embedding and writing to a vector database is a separate step your own pipeline still has to do, which is why Paragon describes Managed Sync as the beginning of your RAG pipeline rather than the whole thing. Read the Managed Sync docs to see the Sync API and Permissions API in detail.