# Data Fundamentals

Sharing my notes from the Data Fundamentals course so that anyone interested can learn along and find it useful :)

## Data Formats

1. **Structured**
    
    Structured data has a fixed schema and its entries are typically tabular. Think, SQL tables.
    
2. **Semistructured**
    
    Data has some structure but the entities can vary from instance to instance. For example, if you’re creating two objects - a chair and a stove, then the objects would have common characteristics like dimension, weight, and material. But it can have different attributes like number of burners, stovetop finishing for stoves and type of seating, cushion type for chair. Common representations are JSON and XML.
    
3. **Unstructured**
    
    Unstructured data is anything that does not have expected attributes. For example, video data, documents, audio, etc, that can vary from instance to instance.
    

## File Formats

**Delimited text files**

Storing data in plain text format with a delimiter or row terminator. Common formats are comma-separated values (CSV) and tab-separated values (TSV)

```plaintext
firstname, lastname, age
allie, james, 12
manny, mohid, 21
```

**JavaScript Object Notation (JSON)**

JSON is a hierarchical document schema used to define objects, and it is flexible. It is versatile and can be used for both structured and semi-structured data.

```json
{
"customers": 
    [
        {
            "firstname": "Mumtaz",
            "major": "Computer Science",
            "contact": [
                        "email": "hello@hello.com"
                        ]
        },
        {
            "firstname": "Janette",
            "major": "Economics",
            "contact": [
                        "email": "jane@hello.com"
                        ]
        },
        }
    ]
}
```

**XML - Extensible Markup Language**

XML uses tags to define elements and attributes. It’s a more readable version of JSON (in my opinion), but it is somewhat harder to parse.

```xml
<Customers> 
    <Customer firstname= "Mumtaz">
        <Major name = "Computer Science>
        <Contact type = "email" = "hello@hello.com">
    </Customer>
    
     <Customer firstname= "Janette">
        <Major name = "Economics">
        <Contact type = "email" = "jane@hello.com">
    </Customer>
 </Customers>
```

**Binary Large Object (BLOB)**

BLOB stores data in binary format and typically needs an application to render the binary files into a human-understandable form. Most audio, video, images, and other media are stored in this format.

**Other formats**

Other formats include columnar and row-wise data stores that enable fast computation. For example, Avro is a row-based format created by Apache that stores the header in JSON format and data in binary. Columnar data formats like ORC or Parquet are also used to quickly search for min, max, or other stats related to the data.

## Data Processing

**Online Transactional Processing (OLTP)**

A transactional system records a transaction that represents a specific event. For example, a debit card processing system would process transactions of money withdrawal, or a point-of-sale system at a major retailer that records each customer's purchase.

Since OLTP manages transactions, the database needs to maintain the integrity of the data recorded. To do so, an important concept in databases is ACID

A - Atomicity  
C - Consistency  
I - Isolation  
D - Durability

**Online Analytical Processing (OLAP)**

OLAP is used for quick processing of queries. Data is stored in a cube format or tabular abstractions. The data is already aggregated, so it is quicker to summarize and get insights from the data.

OLAP takes advantage of data lakes, data warehouses, and data lakehouses. Data lakes store data in file-based data format; data warehouses store data in a relational schema optimized for read operations; and data lakehouses are an optimized version that allow the scalability of a data lake and the querying capabilities of a data lakehouse.