Launching Managed Sync

Today we're excited to unveil Managed Sync - our first-class product for fully managed data ingestion & 3rd-party data access control.

The Challenge: Data Ingestion & Access Control

Integrations can take many forms, such as one off actions (ie. searching for a contact in a CRM), and lightweight workflow automations (ie. sending a Slack notification when an event occurs in your app). But by far the hardest use case to implement is syncing large volumes of data from third-party sources, and keeping that sync up to date.

Building integrations that ingest your users’ external data requires your team to:

1. Build a robust ingestion pipeline to handle hundreds of thousands, if not millions of records, without tripping 3rd-party rate limits and handling any sync failures that occur

2. Repeat that process for every object in an integration, and for every integration you need to support

3. Transform the schema of every 3rd-party object to fit your own schema

4. Build in scheduled jobs to check for updates on a necessary cadence

5. Stay on top of any 3rd-party breaking changes that may impact your pipelines

And if you’re ingesting data for RAG, one additional challenge emerges - the challenge of replicating and enforcing third-party permissions to ensure proper data access control (we wrote an entire tutorial on how to do this yourself).

Managed Sync solves both problems with a purpose-built solution designed for performance, reliability, and ease of implementation.

Overview of Managed Sync

Managed Sync handles all the plumbing and schema normalization with two easy to use APIs, abstracting away all of the challenges above for your team.

The Sync API

With the Sync API, deploying new pipelines that backfill all historical data and refresh on a scheduler only takes a single API call - you won’t need to worry about any of the underlying ingestion logic or infrastructure.

curl --request POST \
  --url https://sync.useparagon.com/api/syncs \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{
  "integration": "googledrive",
  "pipeline": "files",
  "configuration": {
    "frequency" : "1_minute"
  }
}'

Not only that, the ingested data from each source in a category, such as CRM or file storage, will be returned in a normalized format. This makes it extremely easy for you to support every integration in a category.

Learn more about the Sync API in the docs here.

The Permissions API

Designed specifically for RAG ingestion use cases, the Permissions API makes it easy to check a user’s access to a set of data. At a high level:

• Paragon will ingest, index, and maintain permissions from original sources in a fully managed FGA graph, and keep it up to date

• Checking access is extremely easy - just send the document/chunk_ID and user_id

You can learn more about the Permissions API in the docs here.

Getting Started

Managed Sync is now generally available as part of Paragon 2.0. To explore how it can help you rapidly scale your AI product's connectors, check out the Sync playground, book a demo with our team for a full walkthrough, or sign up and try it out for yourself!