# How Rox Handles Refresh Flow from CRM

[warehouse-native-structured-private-data](https://docs.rox.com/development/engineering/docs/system-of-context/warehouse-native-structured-private-data "mention") describes the basic concepts of Rox Data Fabric, and this blog unpacks how Rox refreshes data to give the customers the up-to-date view from their CRM system.

## Why Refresh Flow Matters?

A Customer Relationship Management (CRM) system is not static - customers are constantly adding new records, and updating or deleting existing ones. And customers expect a single source of truth in Rox to view the latest changes in a timely manner. Without a refresh flow:

* Sales teams risk acting on stale leads and losing deals.
* Outreach may mis key contact information to reach out.
* Analytics and forecasting becomes inaccurate, harming strategic decisions.

## Why Refresh Flow is Hard?

Designing a resilient refresh flow involves several engineering challenges:

* High data volume and velocity
* Data Consistency
* Idempotent operations
* Fault Tolerant and Recovery 

## Refresh Flow Architecture

Rox CRM Refresh Flow consists of 3 major layers:

### Data Ingestion Layer

* Connectors to customer CRM detect changes and stream in data changes.

### Processing & Transformation Layer

* Change detection: Detect changes in Account, Person, and Deal entities from CRM based on mapped `crm_last_modified_timestamp`.
* Preprocessing: Cleaning and deduplication of changed data
* Persistence: Refresh changed data in Knowledge Graph

### Serving Layer

* For new data or entity name changes, entity registration would flow into the Rox.
* For key changes in an entity (i.e., domain in Account, email in Person), Rox treats them as a new entity record, and would migrate existing user-added Artifacts to the new entity in Rox.

## Account Refresh

For Account entity, `domain` in `Account` CRM table is the key field for a Rox account.

A Rox account is associated with both

* public data (collected and generated based on `domain`)
* private data for user-added artifacts, including notes, user-uploaded files, contacts, calendars, emails, and outreach campaigns.

For a new CRM account record, Rox would generate a `rox_entity_id` as the unique Rox Account ID. And later account registration kicks in, and will associated with the account owner.

For a domain change on an existing Rox account, Rox would generate a new Rox Account ID as a new account. And later during account registration, Rox would migrate existing private data (aka user artifacts) to the new Rox Account.

For the rest of changed fields, Rox would reflect changes during the refresh, and no Account migration needed.

## Person Refresh

For Person entity, Rox extracts several metadata fields from `Contact` CRM table, including

*  `email` is used to identify a Rox Person.
* `crm_account_id` 

Similarly, for a new CRM person record, Rox would generate a `rox_entity_id` as the unique Rox Person ID, and then will be associated with a Rox account that maps the given `crm_account_id`.

If the mapped `crm_account_id` changed, as part of Account Refresh above, the Account-Person entity link would also be refreshed to reflect that.

For the `email` change, a new Rox Person ID would be generated, and Rox has built unified person entity resolution engine to consolidate matching multiple Rox person into the same unified person (stay tuned for this).