{ "name": "dealwithdata", "generated": "2026-10-08T05:14:21+00:00", "schema": "https://dealwithdata.com/schema/item.schema.json", "count": 22, "items": [ { "id": "dwd_2p63wxfx", "type": "concept", "slug": "business-term-vs-business-element", "title": "Business term vs business element", "summary": "A term is a word and its meaning. An element is governed data that carries that meaning in one context. Governance attaches to the element.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "business", "enterprise" ], "tags": [ "fundamentals", "glossary" ], "terms": [ "dwd_gdefyuzz", "dwd_tthoszn7", "dwd_xzf7arim", "dwd_zx3rfw43", "dwd_gubdecks" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/c/business-term-vs-business-element/", "markdown_url": "https://dealwithdata.com/c/business-term-vs-business-element.md", "body_markdown": "A **business term** is a word and its meaning. A **business element** is a\npiece of data that carries that meaning in a specific business context.\n\n| | Business term | Business element |\n|---|---|---|\n| Lives in | [[business-glossary]] | Business data dictionary or domain |\n| Answers | \"What does this word mean?\" | \"What data do we hold for it, and where?\" |\n| Has | Definition, synonyms, owner | Type, format, allowed values, quality rules, CDE flag |\n| Links to | Elements (one term \u2192 many) | Physical columns (one element \u2192 many) |\n\n## The chain\n\n```\nTerm (meaning) Outstanding Balance\n \u2514\u2500 Element (data) Mortgage Outstanding Balance \u2190 CDE\n \u2502 \u251c\u2500 Column LOAN_DTL.OUTSTANDING_BAL_AMT\n \u2502 \u2514\u2500 Column MTG_HIST.UPB\n \u2514\u2500 Element (data) Card Outstanding Balance\n \u2514\u2500 Column CARD_ACCT.CUR_BAL\n```\n\n## Why the distinction matters\n\n- **Governance attaches to the element.** Stewards,\n [[critical-data-element]] status and quality rules belong to *Mortgage\n Outstanding Balance*, not to the abstract word.\n- **Descriptions belong on the element.** \"For mortgages, excludes escrow\"\n is true of one element and false of the term in general. When AI drafts\n long descriptions, it should draft them per element, grounded in that\n element's columns, profile and lineage.\n- **Mapping is two hops, not one.** Semantic discovery tools propose\n column \u2192 element \u2192 term. Each hop is a proposal with a confidence score\n until a steward approves it.\n\n## Common mistakes\n\n1. **Making every column a term.** The glossary bloats with\n `OUTSTANDING_BAL_AMT_2` and nobody trusts it.\n2. **Skipping elements.** Terms link straight to columns, so there is\n nowhere to hang domain-specific rules or criticality.\n3. **One element per system.** If three systems hold the same mortgage\n balance, that is one element with three columns, not three elements.\n" }, { "id": "dwd_wg5zzqg2", "type": "concept", "slug": "metadata-map", "title": "The metadata map: every piece, one example", "summary": "Catalog, glossary, dictionary, lineage, profiling, quality, contracts, semantic layer: what each one is, shown on a single loan-balance column.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "tags": [ "fundamentals", "metadata" ], "terms": [ "dwd_gubdecks", "dwd_6azimhas", "dwd_lk4rq6xm", "dwd_xzf7arim", "dwd_6prph2yy", "dwd_zx3rfw43", "dwd_2agj6wti", "dwd_2xjo4xi3", "dwd_gxekubnp", "dwd_qywin2zi", "dwd_5k2ibrbv", "dwd_p6v7pmhk", "dwd_qtwdi2p5" ], "related": [ "dwd_2p63wxfx", "dwd_5hhpphgh", "dwd_cb2bgvae" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/c/metadata-map/", "markdown_url": "https://dealwithdata.com/c/metadata-map.md", "body_markdown": "Governance vocabulary is a pile of overlapping words. The fastest way to\nuntangle it is to hold one piece of data still and look at it through each\nlens in turn.\n\n**The example:** a lending table `LOAN_DTL` with a column `OUTSTANDING_BAL_AMT`.\n\n## Every piece, on one column\n\n| Piece | What it is | On our column |\n|---|---|---|\n| [[technical-metadata]] | Facts about the physical data | `LOAN_DTL.OUTSTANDING_BAL_AMT`, `DECIMAL(18,2)`, Oracle, schema `LND` |\n| [[data-dictionary]] | One system's column-by-column reference | \"Unpaid principal, USD, \u2265 0\" |\n| [[business-glossary]] | The enterprise's words and meanings | *Outstanding Balance*: unpaid principal as of the reporting date |\n| [[business-element]] | Governed data carrying that meaning in context | *Mortgage Outstanding Balance* |\n| [[critical-data-element]] | A flag: this one matters most | Yes, because it feeds regulatory reporting |\n| [[data-catalog]] | Searchable inventory of everything | Search \"loan balance\" \u2192 this column, its owner, its reports |\n| [[data-lineage]] | Where it comes from and goes | Core banking \u2192 batch job \u2192 `LOAN_DTL` \u2192 exposure report |\n| [[data-profiling]] | What the data actually looks like | 12M rows, min 0, max 4.2M, 0.3% null |\n| [[data-quality]] | Rules it must pass | \"\u2265 0 and not null\": 99.7% pass |\n| [[data-contract]] | Producer's promise to consumers | Stays `DECIMAL`, lands by 06:00, nulls under 1% |\n| [[reference-data]] | Shared code lists | Status codes `AC` / `CL` / `CO` on the same table |\n| [[master-data]] | The golden version of an entity | The loan's borrower resolves to Customer #123 |\n| [[semantic-layer]] | Calculations defined once | `Total Exposure = SUM(OUTSTANDING_BAL_AMT)` |\n\n## How they connect\n\n1. **Harvest** the [[technical-metadata]] and the source [[data-dictionary]].\n2. **Map** columns to [[business-element]]s and [[business-term]]s. This is\n the business \u2194 technical link, and it is the real product.\n3. **Govern** the elements: owners, [[critical-data-element]] flags, policies,\n [[data-quality]] rules.\n4. **Trace** them with [[data-lineage]].\n5. **Package** them as data products, contracts or a [[semantic-layer]].\n6. **Serve** them to people, systems and AI agents.\n\n## One line each\n\n- **Catalog:** what exists.\n- **Glossary:** what it means.\n- **Dictionary:** what it means *here*.\n- **Lineage:** where it flows.\n- **Profiling:** what it is.\n- **Quality:** whether it's what it should be.\n- **Contract:** what was promised.\n- **Semantic layer:** how to calculate it.\n\n## Three scales\n\n- **Personal:** your spreadsheet of accounts has columns (technical\n metadata), you know what \"balance\" means (glossary), and you'd notice if a\n number went negative (data quality). You're already doing this informally.\n- **Business:** a 20-person company's CRM and accounting system disagree on\n what an \"active customer\" is. A one-page glossary and two quality rules\n fix more than any tool purchase.\n- **Enterprise:** millions of columns across hundreds of systems. None of\n this works by hand; the mapping is proposed by machines and approved by\n people.\n" }, { "id": "dwd_mgglmnwq", "type": "term", "slug": "active-metadata", "title": "Active metadata", "summary": "Metadata that triggers action as it changes, through events, rules and agents, instead of waiting to be looked up.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "related": [ "dwd_cb2bgvae", "dwd_q3gbjug4" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/active-metadata/", "markdown_url": "https://dealwithdata.com/glossary/active-metadata.md", "body_markdown": "Passive metadata is documentation: someone has to go look. Active metadata emits events when it changes, and other systems subscribe and act: block a deploy that breaks a critical element, alert a steward, update an AI agent's context, reassign ownership.\n\n**Test:** if nothing breaks when the metadata is wrong, it isn't active yet.\n" }, { "id": "dwd_tthoszn7", "type": "term", "slug": "business-element", "title": "Business element", "summary": "A governed piece of business data that carries a term's meaning in a specific context, linked to physical columns.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "data element", "business data element" ], "broader": [ "dwd_6azimhas" ], "related": [ "dwd_2p63wxfx" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/business-element/", "markdown_url": "https://dealwithdata.com/glossary/business-element.md", "body_markdown": "A business element answers *what data do we hold for this, and where?* It has a type, format, allowed values, quality rules and often a criticality flag, and it links down to one or more physical columns. Governance (stewards, [[critical-data-element]] status, [[data-quality]] rules) usually attaches here, not to the term.\n\n**Example:** the term *Outstanding Balance* is realized by the elements *Mortgage Outstanding Balance* (a critical data element) and *Card Outstanding Balance*. The mortgage element maps to `LOAN_DTL.OUTSTANDING_BAL_AMT` and `MTG_HIST.UPB`.\n\n**Not to be confused with** a [[business-term]], which is the meaning without the data.\n" }, { "id": "dwd_xzf7arim", "type": "term", "slug": "business-glossary", "title": "Business glossary", "summary": "The organization's agreed vocabulary: business terms, their definitions, synonyms and relationships.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "glossary" ], "broader": [ "dwd_6azimhas" ], "related": [ "dwd_wg5zzqg2", "dwd_2p63wxfx" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/business-glossary/", "markdown_url": "https://dealwithdata.com/glossary/business-glossary.md", "body_markdown": "A business glossary holds the words the business uses and what each one means, independent of any system. It is where \"Customer\", \"Active Account\" and \"Outstanding Balance\" get one agreed definition each, with synonyms and relationships between terms.\n\n**Example:** *Outstanding Balance*: unpaid principal owed as of a date. Synonyms: UPB, principal balance.\n\n**Not to be confused with** a [[data-dictionary]] (one system's columns) or a [[data-catalog]] (the inventory of everything).\n" }, { "id": "dwd_6azimhas", "type": "term", "slug": "business-metadata", "title": "Business metadata", "summary": "What data means to the business: definitions, owners, rules, and how important it is.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "related": [ "dwd_wg5zzqg2", "dwd_2p63wxfx" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/business-metadata/", "markdown_url": "https://dealwithdata.com/glossary/business-metadata.md", "body_markdown": "Business metadata is the human meaning layered on top of [[technical-metadata]]: business terms and their definitions, business elements, owners and stewards, criticality flags, policies and rules. Unlike technical metadata it can rarely be harvested; someone has to decide it, or a tool has to propose it and someone has to approve it.\n\n**Example:** \"Outstanding Balance: the unpaid principal owed on a loan as of the reporting date. Owner: Lending. Critical data element: yes.\"\n\n**Not to be confused with** [[technical-metadata]], which says where data *is*.\n" }, { "id": "dwd_gdefyuzz", "type": "term", "slug": "business-term", "title": "Business term", "summary": "A word or phrase the business uses, with one agreed definition. It describes meaning, not data.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "broader": [ "dwd_6azimhas" ], "related": [ "dwd_2p63wxfx" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/business-term/", "markdown_url": "https://dealwithdata.com/glossary/business-term.md", "body_markdown": "A business term lives in the [[business-glossary]] and answers *what does this word mean?* It has a definition, synonyms and an owner, but no data type and no physical location. One term is usually realized by several [[business-element]]s.\n\n**Example:** *Outstanding Balance*: the unpaid principal owed as of a date.\n\n**Not to be confused with** a [[business-element]], which is a governed piece of data that carries the term's meaning in one context.\n" }, { "id": "dwd_xbqpgs7h", "type": "term", "slug": "change-data-capture", "title": "Change data capture (CDC)", "summary": "Capturing inserts, updates and deletes from a database as a stream of change events, usually by reading its transaction log.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "CDC" ], "related": [ "dwd_cb2bgvae" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/change-data-capture/", "markdown_url": "https://dealwithdata.com/glossary/change-data-capture.md", "body_markdown": "CDC reads a database's transaction log and publishes each row change as an event, typically to Kafka. Tools include Oracle GoldenGate, Debezium, Qlik Replicate and Striim.\n\n**Caution:** raw CDC from a tool's internal database exposes that tool's private schema. Translate row changes into domain events before anyone else subscribes. See the consumption pattern.\n" }, { "id": "dwd_zx3rfw43", "type": "term", "slug": "critical-data-element", "title": "Critical data element (CDE)", "summary": "A data element important enough to the business to require extra governance: an owner, quality rules, documented lineage and monitoring.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "CDE" ], "broader": [ "dwd_tthoszn7" ], "related": [ "dwd_wg5zzqg2", "dwd_2p63wxfx" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/critical-data-element/", "markdown_url": "https://dealwithdata.com/glossary/critical-data-element.md", "body_markdown": "A CDE is a [[business-element]] flagged as critical, usually because it feeds regulatory reports, financial statements or key risk decisions. The flag makes governance mandatory: a named owner, defined [[data-quality]] rules, documented [[data-lineage]] and active monitoring.\n\n**Example:** *Mortgage Outstanding Balance* feeds capital and liquidity reporting, so it is a CDE.\n\n**Three scales**\n- *Personal:* the account numbers and balances your tax return depends on.\n- *Business:* revenue and customer counts in your board report.\n- *Enterprise:* the few thousand fields that regulators and auditors trace end to end.\n" }, { "id": "dwd_lk4rq6xm", "type": "term", "slug": "data-catalog", "title": "Data catalog", "summary": "A searchable inventory of data assets across systems, pointing to their metadata, owners and lineage.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "metadata catalog" ], "broader": [ "dwd_gubdecks" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/data-catalog/", "markdown_url": "https://dealwithdata.com/glossary/data-catalog.md", "body_markdown": "A data catalog answers *what data exists, and where?* It indexes datasets and fields across many systems and connects each to its [[business-glossary]] terms, owners, [[data-lineage]] and quality. People use it to find data; increasingly, AI agents use it to ground themselves.\n\n**Example:** searching \"loan balance\" returns `LOAN_DTL.OUTSTANDING_BAL_AMT`, its definition, its owner, and the reports it feeds.\n\n**Not to be confused with** a [[data-dictionary]], which documents one system in depth.\n" }, { "id": "dwd_qywin2zi", "type": "term", "slug": "data-contract", "title": "Data contract", "summary": "An explicit, versioned agreement between a data producer and its consumers about schema, meaning, quality and freshness.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "related": [ "dwd_cb2bgvae" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/data-contract/", "markdown_url": "https://dealwithdata.com/glossary/data-contract.md", "body_markdown": "A data contract turns unspoken expectations into something checkable: the schema, the semantics, quality thresholds, freshness and who to call. It is enforced in CI or at load time. The Open Data Contract Standard (ODCS) is the leading open format.\n\n**Example:** \"`OUTSTANDING_BAL_AMT` stays `DECIMAL(18,2)`, refreshes daily by 06:00, nulls under 1%.\"\n\n*This site applies the idea to itself:* every content file must satisfy `schema/item.schema.json` before it is published.\n" }, { "id": "dwd_6prph2yy", "type": "term", "slug": "data-dictionary", "title": "Data dictionary", "summary": "A column-by-column reference for one system: each field's type, allowed values and local definition.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "source data dictionary" ], "broader": [ "dwd_gubdecks" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/data-dictionary/", "markdown_url": "https://dealwithdata.com/glossary/data-dictionary.md", "body_markdown": "A data dictionary documents a single source system from its own point of view: every table and field, its type, its allowed values, and what it means *in that system*.\n\n**Example**\n\n| Column | Type | Definition | Allowed values |\n|---|---|---|---|\n| `OUTSTANDING_BAL_AMT` | DECIMAL(18,2) | Unpaid principal, USD | \u2265 0 |\n| `LOAN_STAT_CD` | CHAR(2) | Loan status | `AC` active, `CL` closed, `CO` charged off |\n\n**Not to be confused with** a [[business-glossary]], which defines terms for the whole enterprise. Linking dictionary entries to glossary terms is the business-to-technical mapping.\n" }, { "id": "dwd_2agj6wti", "type": "term", "slug": "data-lineage", "title": "Data lineage", "summary": "The record of where data comes from, how it is transformed, and where it goes.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "lineage" ], "broader": [ "dwd_gubdecks" ], "related": [ "dwd_wg5zzqg2", "dwd_cb2bgvae" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/data-lineage/", "markdown_url": "https://dealwithdata.com/glossary/data-lineage.md", "body_markdown": "Lineage traces data from its origin through every transformation to every consumer. *Upstream* lineage explains where a number came from; *downstream* lineage shows what breaks if it changes. OpenLineage is the open standard for emitting it from pipelines at run time.\n\n**Example:** core banking system \u2192 nightly batch job \u2192 `LOAN_DTL.OUTSTANDING_BAL_AMT` \u2192 exposure report.\n" }, { "id": "dwd_2xjo4xi3", "type": "term", "slug": "data-profiling", "title": "Data profiling", "summary": "Measuring what data actually looks like: counts, ranges, nulls, distinct values and patterns.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "profiling" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/data-profiling/", "markdown_url": "https://dealwithdata.com/glossary/data-profiling.md", "body_markdown": "Profiling describes data as it *is*: row counts, minimums and maximums, null rates, distinct values and value patterns. It is the evidence base for [[data-quality]] rules and a strong input for AI that suggests business meaning.\n\n**Example:** `OUTSTANDING_BAL_AMT`: 12M rows, min 0, max 4.2M, 0.3% null.\n\n**Not to be confused with** [[data-quality]], which checks data against what it *should* be.\n" }, { "id": "dwd_gxekubnp", "type": "term", "slug": "data-quality", "title": "Data quality", "summary": "Rules data must satisfy, and the measured results of checking it against them.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "DQ" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/data-quality/", "markdown_url": "https://dealwithdata.com/glossary/data-quality.md", "body_markdown": "Data quality is the gap between what data should be and what it is. Rules (\"never negative\", \"not null\", \"matches the reference list\") are checked, scored and monitored. Active data quality blocks or flags bad data *before* it reaches a critical report.\n\n**Example:** \"Outstanding balance must be \u2265 0 and not null\": 99.7% pass this run.\n\n**Not to be confused with** [[data-profiling]], which only describes.\n" }, { "id": "dwd_qtwdi2p5", "type": "term", "slug": "master-data", "title": "Master data", "summary": "The single trusted version of a core business entity such as a customer, product or account.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "golden record" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/master-data/", "markdown_url": "https://dealwithdata.com/glossary/master-data.md", "body_markdown": "Master data management produces one golden record for each core entity by matching and merging copies from many systems. It answers *which* customer, while [[reference-data]] answers *which code*.\n\n**Example:** five systems hold variations of the same customer; master data resolves them to Customer #123.\n" }, { "id": "dwd_p6v7pmhk", "type": "term", "slug": "reference-data", "title": "Reference data", "summary": "Shared lists of allowed codes and values used across systems.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/reference-data/", "markdown_url": "https://dealwithdata.com/glossary/reference-data.md", "body_markdown": "Reference data is the set of allowed values that many systems share: country codes, currency codes, status codes, product categories. It changes rarely, matters everywhere, and causes quiet breakage when systems disagree.\n\n**Example:** loan status codes `AC` (active), `CL` (closed), `CO` (charged off).\n" }, { "id": "dwd_5k2ibrbv", "type": "term", "slug": "semantic-layer", "title": "Semantic layer", "summary": "A shared definition of business metrics and dimensions, so every tool and AI agent calculates them the same way.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "metrics layer" ], "related": [ "dwd_5hhpphgh" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/semantic-layer/", "markdown_url": "https://dealwithdata.com/glossary/semantic-layer.md", "body_markdown": "A semantic layer defines calculations once (metrics, dimensions, joins) over physical data, and every BI tool and AI agent queries through it. The Open Semantic Interchange (OSI), now Apache Ossie, is the emerging format for exchanging these definitions between tools.\n\n**Example:** `Total Exposure = SUM(OUTSTANDING_BAL_AMT)`, synonyms \"UPB\", \"principal\".\n\n**Not to be confused with** *semantic discovery*, a catalog activity that infers what a column *means*. A semantic layer defines how to *calculate*.\n" }, { "id": "dwd_gubdecks", "type": "term", "slug": "technical-metadata", "title": "Technical metadata", "summary": "Facts about how data is physically stored and moved: names, types, locations, schemas and jobs.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "personal", "business", "enterprise" ], "synonyms": [ "physical metadata" ], "related": [ "dwd_wg5zzqg2" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/glossary/technical-metadata/", "markdown_url": "https://dealwithdata.com/glossary/technical-metadata.md", "body_markdown": "Technical metadata describes the data as the machines see it: database, schema, table and column names, data types, keys, file paths, and the jobs that read and write them. It is usually harvested automatically from the systems themselves.\n\n**Example:** `LOAN_DTL.OUTSTANDING_BAL_AMT`, `DECIMAL(18,2)`, in schema `LND` on an Oracle database, loaded nightly by a batch job.\n\n**Three scales**\n- *Personal:* the folder, file name and format of your tax spreadsheet.\n- *Business:* the columns in your accounting system's invoice table.\n- *Enterprise:* millions of datasets harvested from hundreds of platforms.\n\n**Not to be confused with** [[business-metadata]], which says what the data *means*.\n" }, { "id": "dwd_cb2bgvae", "type": "pattern", "slug": "active-metadata-consumption", "title": "Serving metadata to people, systems and agents", "summary": "One canonical model, one governed write path, many read paths. How to turn catalog changes into subscribable domain events, and which channel serves which consumer.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "enterprise" ], "tags": [ "architecture", "events", "ai", "mcp", "kafka" ], "terms": [ "dwd_mgglmnwq", "dwd_xbqpgs7h", "dwd_2agj6wti", "dwd_qywin2zi", "dwd_5k2ibrbv", "dwd_tthoszn7", "dwd_gdefyuzz" ], "related": [ "dwd_q3gbjug4", "dwd_5hhpphgh", "dwd_wg5zzqg2" ], "sources": [ { "title": "OpenLineage", "url": "https://openlineage.io/", "accessed": "2026-10-07" }, { "title": "CloudEvents specification", "url": "https://cloudevents.io/", "accessed": "2026-10-07" }, { "title": "AsyncAPI", "url": "https://www.asyncapi.com/", "accessed": "2026-10-07" }, { "title": "Model Context Protocol", "url": "https://modelcontextprotocol.io/", "accessed": "2026-10-07" }, { "title": "Open Data Contract Standard (Bitol)", "url": "https://bitol-io.github.io/open-data-contract-standard/", "accessed": "2026-10-07" } ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/c/active-metadata-consumption/", "markdown_url": "https://dealwithdata.com/c/active-metadata-consumption.md", "body_markdown": "Once a catalog has good business metadata, everyone asks for it: \"Can we\nget technical metadata with business metadata?\", \"Can our AI agent use the\ndefinitions?\" The failure mode is ten teams building ten extracts. The fix\nis a shape, not a tool.\n\n## The shape\n\n**One canonical model, one governed write path, many read paths.**\n\nThe hard, valuable part isn't the streaming. It's the governed link:\n[[business-term]] \u2192 [[business-element]] \u2192 physical column \u2192 lineage.\nEverything below is delivery.\n\n## Getting changes out\n\nMost catalogs sit on a relational database and are poll or harvest based.\nChanges arrive two ways:\n\n- **UI edits** (a steward changes a definition): capture with database\n [[change-data-capture]] such as GoldenGate or Debezium.\n- **Harvests** (scanners reload technical metadata): diff successive\n snapshots and emit a change set. This gives cleaner events than the\n database log.\n\n## Three layers, not one\n\n```\nCatalog DB \u2500\u2500CDC\u2500\u2500\u25b6 raw.* topics (private: one team reads these)\nHarvests \u2500\u2500diff\u2500\u2500\u25b6 \u2502\n \u25bc\n Translator row changes \u2192 domain events\n \u2502 schema registry, versioned\n \u25bc\n public topics BusinessElementDefinitionChanged\n LineageEdgeAdded\n CdeStewardReassigned\n```\n\nNever expose raw CDC from a vendor's internal schema. It is undocumented,\nchanges on upgrade, and one business edit fires a dozen row changes.\nSubscribers break and drown.\n\n## Pick the channel by consumer\n\n| Consumer | Channel | Standard |\n|---|---|---|\n| Stewards, analysts | Catalog UI | n/a |\n| Applications | REST or GraphQL over the metadata graph | OpenAPI |\n| Change subscribers | Domain-event topics | CloudEvents envelope, AsyncAPI docs |\n| Bulk analytics | Metadata snapshots as tables | SQL |\n| AI agents | MCP server plus a search index | MCP |\n| Pipelines | Emit lineage at run time; enforce contracts in CI | OpenLineage, ODCS |\n| BI and AI tools | Semantic model export | OSI (Apache Ossie) |\n\nAll of these are projections of the same model. When a team asks for \"a\nfeed\", they get one of these, not a custom extract.\n\n## Rules that keep it trustworthy\n\n1. **Stable global IDs** for every term, element and physical asset, or the\n links can't survive across systems.\n2. **The link is a governed object.** A business-to-technical mapping carries\n an owner, a confidence score, provenance (human, rule or AI) and a status.\n AI-proposed mappings enter as *proposed*; stewards authorize.\n3. **The catalog stays the system of record for its domain.** Publish\n outward; don't try to make it the hub for everything.\n4. **Agents read through the MCP layer, never the database.** That gives one\n place for entitlements, audit, and \"approved definitions only\".\n\n## When it becomes a nervous system\n\n[[active-metadata]] only matters if something depends on it. Wire in two or\nthree consumers that break when metadata is wrong, such as change\nmanagement, a regulatory report pipeline, and one AI agent. Until then,\nit's documentation.\n\n## Using LLMs for long descriptions\n\nFor bulk drafting of element descriptions, a fast, cheap model is the right\ndefault. Quality comes from grounding, not model size: feed the physical\nname and type, profile statistics, lineage neighbors, the parent dataset\nand nearby glossary terms. Route critical elements and low-confidence drafts\nto a stronger model or a human reviewer.\n" }, { "id": "dwd_q3gbjug4", "type": "pattern", "slug": "incremental-metadata-feeds", "title": "Never recompute everything: incremental metadata feeds", "summary": "A feed that reprocesses millions of datasets every run stops scaling. Recompute only what changed, plus what changed rules touch, and reconcile on a schedule.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "business", "enterprise" ], "tags": [ "performance", "architecture", "harvest" ], "terms": [ "dwd_mgglmnwq", "dwd_gubdecks", "dwd_tthoszn7" ], "related": [ "dwd_cb2bgvae" ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/c/incremental-metadata-feeds/", "markdown_url": "https://dealwithdata.com/c/incremental-metadata-feeds.md", "body_markdown": "**The situation:** a feed assigns every harvested dataset to an owning\napplication. At a few thousand datasets it runs in minutes. At millions of\ndatasets and tens of millions of elements, it runs for hours, because it\nrecomputes and reloads everything, every run.\n\n## Find where the time goes\n\nSplit a run into three phases and time each:\n\n1. **Extract:** read datasets and elements out of the catalog.\n2. **Compute:** match each dataset to an application (paths, hosts, naming rules).\n3. **Load:** write assignments back.\n\nLoad is usually the worst. Catalog imports tend to be row by row with\nversioning and audit, so rewriting a million unchanged rows costs nearly as\nmuch as writing changed ones.\n\n## Fixes, biggest first\n\n1. **Go incremental.** Drive the feed from a harvest diff: only added or\n changed datasets are reassigned. If 1\u20132% change per cycle, that's a\n 50\u2013100\u00d7 reduction.\n2. **Assign datasets; let elements inherit.** Touching every element\n individually multiplies the work by the elements-per-dataset ratio.\n3. **Skip no-op writes.** Compare computed to current, and load only the\n differences. This alone often cuts load time by 90%.\n4. **Compute in sets.** Put rules in a table and resolve them with one join\n or prefix match, not a loop of lookups. An embedded engine such as DuckDB\n handles millions of rows in minutes.\n5. **Partition.** Split by application or path prefix so one failure\n doesn't rerun everything.\n\n## Is a full recompute ever needed?\n\nNot every run. Two kinds of change drive work:\n\n- **A dataset changed** \u2192 recompute that dataset.\n- **A rule or reference list changed** (a mapping edited, an application\n split or retired, a host moved) \u2192 recompute only the datasets the old\n *or* new rule matches.\n\nA full pass is justified only for:\n\n- **Bootstrap**, and after major platform or model changes.\n- **Scheduled reconciliation**, weekly or monthly, as a backstop for missed\n diffs, failed runs and direct edits. Compute everything, still write only\n differences, and report drift.\n- **Attestation**, when an auditor needs proof the whole population was\n evaluated at a point in time.\n\nIf drift stays near zero for several cycles, stretch the interval. That's\nyour evidence that incremental is safe.\n\n## The target shape\n\n```\nharvest \u2192 diff \u2192 assign changed (set-based) \u2192 load differences only\n \u2192 emit AssignmentChanged events\n```\n\n**The one thing to get right:** rule changes must emit events too. If\nsomeone edits a mapping table and nothing notices, incremental runs go\nquietly stale. Catching that is what reconciliation is for.\n\n## Three scales\n\n- **Business:** the same logic applies to any nightly sync. Diff first, then\n write only what changed, and your 2-hour job becomes 5 minutes.\n- **Enterprise:** at millions of objects, \"full refresh\" is a design bug,\n not a strategy.\n" }, { "id": "dwd_5hhpphgh", "type": "teardown", "slug": "open-semantic-interchange", "title": "Open Semantic Interchange (Apache Ossie): what it is, and what it isn't yet", "summary": "OSI is a portable format for semantic models: datasets, fields, joins, metrics and AI context. It is not a catalog, glossary, lineage or governance standard. Export to it; don't build your core on it yet.", "status": "draft", "classification": "public", "authors": [ { "name": "Shan Umasankar", "url": "https://dealwithdata.com", "role": "author" } ], "assisted_by": [ "Claude" ], "scales": [ "business", "enterprise" ], "tags": [ "standards", "semantic-layer", "ai" ], "terms": [ "dwd_5k2ibrbv", "dwd_xzf7arim", "dwd_2agj6wti", "dwd_lk4rq6xm" ], "related": [ "dwd_wg5zzqg2", "dwd_cb2bgvae" ], "as_of": "2026-10-07", "sources": [ { "title": "Apache Ossie: Core Metadata Specification", "url": "https://github.com/apache/ossie/blob/main/core-spec/spec.md", "accessed": "2026-10-07" }, { "title": "Apache Ossie Roadmap", "url": "https://github.com/apache/ossie/blob/main/ROADMAP.md", "accessed": "2026-10-07" }, { "title": "OSI Community Update: What's New and What's Next (April 2026)", "url": "https://ossie.apache.org/updates/osi-april-2026-community-update/", "accessed": "2026-10-07" }, { "title": "Snowflake: OSI specification now live", "url": "https://www.snowflake.com/en/blog/open-semantic-interchanges-specs-finalized/", "accessed": "2026-10-07" } ], "created": "2026-10-07", "updated": "2026-10-07", "version": 1, "url": "https://dealwithdata.com/c/open-semantic-interchange/", "markdown_url": "https://dealwithdata.com/c/open-semantic-interchange.md", "body_markdown": "**In one line:** OSI lets BI and AI tools exchange the *structure* of a\n[[semantic-layer]] model. Everything else, such as identity, governance,\nglossary and lineage, is out of scope or still on the roadmap.\n\n## Status\n\n- Started by Snowflake with dbt Labs, Databricks, Salesforce and others; the\n first spec went public in January 2026 under Apache 2.0.\n- Now **Apache Ossie**, in the Apache Incubator. Oracle, Cloudera, Dataiku,\n Denodo and Dremio are among the later joiners.\n- Only formal release: **0.1.1** (December 2025). The in-development\n **0.2.0.dev0** has already made one breaking change: one model per\n document, with the old `semantic_model` array removed. The spec itself\n says not to depend on it in production.\n\n## The whole model\n\n| Construct | What it holds |\n|---|---|\n| Semantic model | `version`, `name`, `description`, `ai_context`, and the lists below |\n| Datasets | Logical entities pointing at a physical `source`, with primary and unique keys |\n| Fields | Row-level attributes defined by an expression, in one or more SQL dialects |\n| Relationships | Foreign-key joins between datasets, simple or composite |\n| Metrics | Aggregate expressions at model level; may span datasets |\n| `ai_context` | Instructions, synonyms and example questions, on almost any object |\n| `custom_extensions` | Vendor-tagged JSON for anything the core doesn't cover |\n\nSupported dialects include ANSI SQL, Snowflake, Databricks, BigQuery, DAX,\nMDX, Tableau and Ossie's own portable SQL.\n\n## What it does not cover (yet)\n\n- **Stable identifiers.** Objects are identified by `name`; stable IDs are an\n open discussion.\n- **Governance.** No owner, steward, certification or criticality field.\n- **Sensitivity.** PII and confidentiality flags are proposals only.\n- **Meaning.** No [[business-glossary]] or ontology; the project describes\n its own scope as structural, not conceptual, interoperability. An ontology\n working group is active.\n- **Lineage.** Use OpenLineage.\n- **Storage and versioning.** No registry and no model versioning. The\n document's `version` is the *spec* version, not yours. Catalog\n integration and a semantic registry are roadmap items.\n\n## A governed field, the practical way\n\nCarry your governance in an extension until the spec grows the slots:\n\n```yaml\n- name: mortgage_outstanding_balance\n expression:\n dialects:\n - dialect: ANSI_SQL\n expression: outstanding_bal_amt\n datatype: Decimal\n description: Unpaid principal on the mortgage as of the reporting date, excluding escrow.\n ai_context:\n synonyms: [\"UPB\", \"outstanding principal\"]\n custom_extensions:\n - vendor_name: ACME_CATALOG\n data: '{\"global_id\":\"BE-000123\",\"cde\":true,\"status\":\"certified\",\"model_version\":\"2026.10.1\"}'\n```\n\n## Where to store and version OSI files\n\nNothing is standardized, so do what works today:\n\n1. **Generate** OSI from your catalog, the system of record. Never hand-edit.\n2. **Keep it in Git**, one file per model. CI runs the project's validator\n (schema, unique names, references, SQL syntax).\n3. **Tag a release** when stewards approve a change set.\n4. **Deploy outward** with the community converters (Snowflake Cortex\n Analyst, GoodData, Salesforce, Apache Polaris).\n\n## Verdict\n\n- **Don't build your core model on it.** It is pre-1.0, has broken its schema\n once, and lacks IDs and governance.\n- **Do export to it.** It is cheap, and the BI and AI vendors are converging\n on it.\n- **Join the working groups if you can.** Governance, identity and catalog\n integration are exactly where regulated industries have the most to say,\n and they are under-represented.\n\n## Three scales\n\n- **Business:** if you run dbt or a single BI tool, OSI is mostly invisible;\n it matters when you add a second tool or an AI analyst.\n- **Enterprise:** OSI is one outbound channel among several, next to\n lineage, events and an agent interface. See the consumption pattern.\n" } ] }