How Rox Handles Refresh Flow from CRM

Warehouse Native Structured Private Data describes the basic concepts of Rox Data Fabric, and this blog unpacks how Rox refreshes data to give the customers the up-to-date view from their CRM system.

Why Refresh Flow Matters?

A Customer Relationship Management (CRM) system is not static - customers are constantly adding new records, and updating or deleting existing ones. And customers expect a single source of truth in Rox to view the latest changes in a timely manner. Without a refresh flow:

• Sales teams risk acting on stale leads and losing deals.

• Outreach may mis key contact information to reach out.

• Analytics and forecasting becomes inaccurate, harming strategic decisions.

Why Refresh Flow is Hard?

Designing a resilient refresh flow involves several engineering challenges:

• High data volume and velocity

• Data Consistency

• Idempotent operations

• Fault Tolerant and Recovery

Refresh Flow Architecture

Rox CRM Refresh Flow consists of 3 major layers:

Data Ingestion Layer

• Connectors to customer CRM detect changes and stream in data changes.

Processing & Transformation Layer

• Change detection: Detect changes in Account, Person, and Deal entities from CRM based on mapped crm_last_modified_timestamp.

• Preprocessing: Cleaning and deduplication of changed data

• Persistence: Refresh changed data in Knowledge Graph

Serving Layer

• For new data or entity name changes, entity registration would flow into the Rox.

• For key changes in an entity (i.e., domain in Account, email in Person), Rox treats them as a new entity record, and would migrate existing user-added Artifacts to the new entity in Rox.

Account Refresh

For Account entity, domain in Account CRM table is the key field for a Rox account.

A Rox account is associated with both

• public data (collected and generated based on domain)

• private data for user-added artifacts, including notes, user-uploaded files, contacts, calendars, emails, and outreach campaigns.

For a new CRM account record, Rox would generate a rox_entity_id as the unique Rox Account ID. And later account registration kicks in, and will associated with the account owner.

For a domain change on an existing Rox account, Rox would generate a new Rox Account ID as a new account. And later during account registration, Rox would migrate existing private data (aka user artifacts) to the new Rox Account.

For the rest of changed fields, Rox would reflect changes during the refresh, and no Account migration needed.

Person Refresh

For Person entity, Rox extracts several metadata fields from Contact CRM table, including

• email is used to identify a Rox Person.

• crm_account_id

Similarly, for a new CRM person record, Rox would generate a rox_entity_id as the unique Rox Person ID, and then will be associated with a Rox account that maps the given crm_account_id.

If the mapped crm_account_id changed, as part of Account Refresh above, the Account-Person entity link would also be refreshed to reflect that.

For the email change, a new Rox Person ID would be generated, and Rox has built unified person entity resolution engine to consolidate matching multiple Rox person into the same unified person (stay tuned for this).