2.2 Data Formats

In this tutorial, we will manipulate the data structure from and to several data formats: JSON, CSV, Parquet, GeoTIFF, GeoJSON, netCDF/HDF5, and Zarr. At the end, we compare the on-disk size of the same data across formats.

All files created or downloaded in this notebook go into a local data/ folder.

🖥️ Lecture slides — Session 04 (Wed Oct 7)

JSON (JavaScript Object Notation)¶

JSON is a lightweight, human-readable data format used in web applications and APIs for data exchange. JSON is used to store metadata, configuration files, and small datasets, particularly when working with web-based applications or interacting with APIs (e.g., querying weather data or geospatial information from an API). Here is a simple example:

{
    "location": "Yellowstone",
    "coordinates": {
        "latitude": 44.423691,
        "longitude": -110.588516
    },
    "elevation_m": 2399,
    "temperature_c": 22.5
}

The character encoding is UTF-8. The data types in JSON files may be numbers, string, boolean, array, object (collection of name-value pairs), or null. More information on JSON from the EarthDataScience course.

In Python, you create a JSON file with the json library:

Tabular Data Formats¶

Tabular data are common in geosciences. The CSV format is a convenient, human-readable format for small data sets that stores data in rows. The Parquet format is machine-readable, columnar, and compressed, built for large data sets.

CSV (Comma-Separated Values):¶

CSV is a simple, widely used format for tabular data, often used in geoscience for sharing and storing smaller datasets (e.g., soil samples, environmental readings). It stores data as plain text, making it easy to read but inefficient for large datasets. It is most often read using pandas.

Parquet:¶

Parquet is a binary, columnar storage format optimized for efficiency, particularly for large datasets. It is widely used in big data environments (e.g., storing satellite imagery or climate model outputs).

Geospatial Data¶

The main formats for geospatial data in this lesson are:

• GeoTIFF: a metadata standard that embeds georeferencing information in a TIFF (Tagged Image File Format) file. Cloud-Optimized GeoTIFF (COG) is a variant organized for remote access.
• GeoJSON: a format for encoding a variety of geographic data structures (points, lines, polygons) in JSON.

1. Raster data¶

1.1 rasterio to read GeoTIFF¶

Raster data is any pixelated (or gridded) data where each pixel is associated with a specific geographical location. The value of a pixel can be continuous (e.g., elevation) or categorical (e.g., land use).

The python package rasterio (documentation) reads formats such as GeoTIFF.

See additional introductory materials from EarthDataScience and tutorials from the GeoHackweek.

We will download a shaded-relief topography raster from Natural Earth. The file name is HYP_50M_SR and it comes as a zipped file. We use pooch to download it into data/ and unzip it in one step.

Now let’s open the GeoTIFF with rasterio.

A rasterio dataset is not a netCDF file: it has no .variables. Instead, it stores one or more bands. The basic attributes are .count (number of bands), .width and .height (raster dimensions), and .crs (coordinate reference system).

Can you guess how to get the data types of the bands?

And the boundaries of the dataset:

How to interpret the data: there are three bands, one for each of the colors red, green, and blue:

elevation.read(1) returns the first band as a 2D numpy array. Let us plot it alone, then all three bands as a color image.

1.2 Geopandas to read GeoJSON¶

GeoJSON is a special case of JSON that stores geographic features (points, lines, polygons) with their attributes. Instead of downloading one, we will build a small GeoJSON from scratch: a FeatureCollection of four Cascade volcanoes. This shows the structure of the format explicitly.

Note the structure: each Feature has a geometry (here a Point with longitude first, then latitude) and a properties dictionary for the attributes. Now read the file back with geopandas:

Geopandas parsed the geometry column and set the coordinate reference system to WGS84 (EPSG:4326), the GeoJSON default:

Let us plot the data on an interactive map. Folium is a nice Python package for visualization. The Geohackweek tutorial on Folium is also informative.

2. Hierarchical formats: NETCDF4 & HDF5¶

Hierarchical data formats are designed to store large amounts of data in a single file. They mimic a file system (a tree-like data structure with nested directories) inside a single file. There are two dominant hierarchical data formats (HDF5 and NETCDF4), and one built for the cloud (Zarr). Hierarchical formats can store many data types (numeric vs string).

HDF5¶

The Hierarchical Data Format version 5 (HDF5) is an open-source file format that supports large, complex, heterogeneous data. HDF5 uses a “file directory” like structure that allows you to organize data within the file in many different structured ways, as you might do with files on your computer. The HDF5 format also allows for embedding of metadata, making it self-describing. All elements (the file itself, groups, and datasets) can have associated metadata that describes the information contained within the element.

HDF Structure Example:

• Datasets, which are typed multidimensional arrays
• Groups, which are container structures that can hold datasets and other groups

NetCDF¶

The network Common Data Form, or netCDF, was created in the early 1990s to solve some of the challenges in working with N-dimensional arrays. Netcdf is a collection of self-describing, machine-independent binary data formats and software tools that facilitate the creation, access, and sharing of scientific data stored in N-dimensional arrays, along with metadata describing the contents of each array. Netcdf was built by the climate science community at a time when regional climate models were beginning to produce larger and larger output files. NetCDF version 4 is a subset of HDF5, so netCDF4 files can be opened with HDF5 tools.

Handling large arrays¶

The netCDF and HDF5 formats have no limit on file size. However, any analysis tool that reads data from a netCDF array into memory for some computational operation is limited by that particular machine’s available memory.

But slow at I/O¶

When reading a hierarchical file, the whole tree of the data structure is scanned from the root node down. Since this has to be done each time a user makes an inquiry, reading HDF5 and netCDF is slow compared to formats designed for parallel or cloud access.

2.1 Create a dataset and write it to netCDF¶

We create a synthetic monthly temperature-anomaly field with the course package mlgeo_synth, keep one time slice as a 2D field, and wrap it in an xarray Dataset with coordinates and units.

2.2 Read it back with xarray¶

2.3 The same file through the netCDF4 and h5py libraries¶

The netCDF4 library shows the self-describing structure: dimensions, variables, and attributes.

Because netCDF4 files are HDF5 files, h5py can open the same file. The variables appear as HDF5 datasets, and the metadata as HDF5 attributes.

3. Zarr¶

Zarr is a cloud-optimized format for N-dimensional arrays. Instead of a single file, a Zarr store is a directory of compressed chunks plus small JSON metadata files. Cloud object stores and parallel jobs can read individual chunks without scanning the whole file tree, which removes the netCDF/HDF5 I/O bottleneck.

Xarray writes to Zarr directly:

The round trip preserves the data and the metadata:

4. Comparing on-disk sizes across formats¶

We now store the same data in the formats above and compare their sizes on disk.

• For the table comparison, we flatten the 2D field to a dataframe (one row per grid cell) and write it to CSV and Parquet.
• For the array comparison, we use the netCDF file and the Zarr store we already wrote. A Zarr store is a directory, so we sum the sizes of all files inside it.

Interpretation: the CSV file is the largest because it stores every number as text, digit by digit. Parquet stores the same table in a compressed binary layout and is several times smaller. The netCDF file and the Zarr store hold the same array in binary form and end up in the same size range; the difference between them at this scale comes from container overhead (HDF5 headers for netCDF, chunk files and JSON metadata for Zarr), not from the data itself. The gap between text and binary formats grows with dataset size, and Zarr’s chunked layout pays off when many processes read different pieces of a large array at once.