Cross-matching of large catalogs: DES to Gaia

Cross-matching of large catalogs: DES to Gaia#

In this tutorial, we will:

retrieve the original DES DR2 and Gaia DR3 data files
convert the data to HATS format using hats-import
cross-match the catalogs with LSDB and save the result as a new HATS catalog

Introduction#

This notebook shows a realistic crossmatching of two different survey datasets, starting at the original data files as hosted by their providers.

DES DR2

The Dark Energy Survey DR2 (DES DR2) catalog is hosted by NCSA, see the official website for more information. Data files, in FITS format, are located at https://desdr-server.ncsa.illinois.edu/despublic/dr2_tiles/. You may also prefer to get the data with S3 client as described by https://desdr-server.ncsa.illinois.edu.

Gaia DR3

The Gaia DR3 catalog is hosted by ESA, see the official website for more information. Data files, in ECSV format are located at http://cdn.gea.esac.esa.int/Gaia/gdr3/gaia_source/.

We use gaia_source table, see its schema here.

1. Install required packages and import modules#

[1]:

# Comment to skip hats-import installation
%pip install --quiet -U hats-import

# Uncomment to install lsdb
%pip install --quiet -U lsdb


[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: pip3 install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

[notice] A new release of pip is available: 24.3.1 -> 25.1.1
[notice] To update, run: pip3 install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

[2]:

# For Gaia columns data types inference
from astropy.io import ascii

# We need this for 2D histogram
import dask.array

# Client for Dask distributed computing
from dask.distributed import Client

# For plots
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm

# Numpy is always useful!
import numpy as np

# For files and directories manipulation
from upath import UPath

# Explore the HATS catalogs and plot sky maps
from hats import read_hats
from hats.inspection import plot_pixels

# For reading the HATS catalogs and performing the cross-match
import lsdb

# For converting the data to HATS format and generate margin caches
from hats_import.catalog.file_readers import CsvReader
from hats_import.margin_cache.margin_cache_arguments import MarginCacheArguments
from hats_import.pipeline import ImportArguments, pipeline_with_client

2. Get original data files#

To make this notebook faster to run and using less storage, we will use a subset of the data. However, the same pipeline, with no code changes, may be used to process the full catalogs.

[3]:

# Change to the directories where the data will be stored
DES_DIR = UPath("data/DES_DR2")
GAIA_DIR = UPath("data/Gaia_DR3")

GAIA_SCHEMA_FILE = GAIA_DIR / "schema.parquet"

DES_HATS_NAME = "des_dr2"
GAIA_HATS_NAME = "gaia_dr3"
GAIA_MARGIN_CACHE_NAME = "gaia_dr3_1arcsec"
XMATCH_NAME = "des_dr2_x_gaia_dr3"

HATS_DIR = UPath("catalogs")
DES_HATS_DIR = HATS_DIR / DES_HATS_NAME
GAIA_HATS_DIR = HATS_DIR / GAIA_HATS_NAME
GAIA_MARGIN_CACHE_DIR = HATS_DIR / GAIA_MARGIN_CACHE_NAME

XMATCH_DIR = HATS_DIR / XMATCH_NAME

2.1. DES DR2#

The Dark Energy Survey DR2 catalog is hosted by NCSA, see the official website for more information. Data files, in FITS format, are located at https://desdr-server.ncsa.illinois.edu/despublic/dr2_tiles/. You may also prefer to get the data with S3 client as described by https://desdr-server.ncsa.illinois.edu.

We use *dr2_main.fits files for the main catalog table; see the schema here.

Here we download a few first files to demonstrate the pipeline. The full catalog is about 1.1TB–feel free to download it if you have enough storage.

[4]:

# Comment / skip this cell if you already have the data

# Download few first files
des_urls = [
    UPath(
        "https://desdr-server.ncsa.illinois.edu/despublic/dr2_tiles/DES0000+0209/DES0000+0209_dr2_main.fits"
    ),
    UPath(
        "https://desdr-server.ncsa.illinois.edu/despublic/dr2_tiles/DES0000+0252/DES0000+0252_dr2_main.fits"
    ),
    UPath(
        "https://desdr-server.ncsa.illinois.edu/despublic/dr2_tiles/DES0000+0335/DES0000+0335_dr2_main.fits"
    ),
]

for des_url in des_urls:
    des_file = DES_DIR / des_url.name
    if not des_file.exists():
        des_file.parent.mkdir(parents=True, exist_ok=True)
        des_file.write_bytes(des_url.read_bytes())

2.2. Gaia DR3#

Here we download a few files which barely correspond to the same area of the sky as the DES DR2 files above. The full catalog is much larger, feel free to download it all if you have enough storage.

[5]:

# Comment / skip this cell if you already have the data

# Download few first files
gaia_urls = [
    UPath("http://cdn.gea.esac.esa.int/Gaia/gdr3/gaia_source/GaiaSource_310878-313367.csv.gz"),
]

for gaia_url in gaia_urls:
    gaia_file = GAIA_DIR / UPath(gaia_url).name
    if not gaia_file.exists():
        gaia_file.parent.mkdir(parents=True, exist_ok=True)
        gaia_file.write_bytes(gaia_url.read_bytes())

3. Create Dask client#

We use Dask for parallel processing.

We create a single Dask client here to be used in all the following pipelines steps (both for the import pipelines and LSDB processing pipelines), and we destroy it at the end of the notebook.

Additional Help

For additional information on dask client creation, please refer to the official Dask documentation and our Dask cluster configuration page for LSDB-specific tips. Note that dask also provides its own best practices, which may also be useful to consult.

For tips on accessing remote data, see our Accessing remote data guide

[6]:

client = Client()

/astro/users/kmalanch/.virtualenvs/default/lib/python3.10/site-packages/distributed/node.py:187: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 37065 instead
  warnings.warn(

4. Convert the data to HATS format#

We use the hats-import tool to create HATS catalogs from the original data files.

4.1. Convert DES DR2 to HATS#

Plan the pipeline specifying all parameters of the conversion
Run the pipeline with Dask

[7]:

des_args = ImportArguments(
    sort_columns="COADD_OBJECT_ID",
    ra_column="RA",
    dec_column="DEC",
    input_path=DES_DIR,
    file_reader="fits",
    output_artifact_name=DES_HATS_NAME,
    output_path=HATS_DIR,
)

pipeline_with_client(des_args, client)

tmp_path (catalogs/des_dr2/intermediate) contains intermediate files; resuming prior progress.

4.2. Plot the DES DR2 HATS catalog pixels#

[8]:

# Read the HATS catalog metadata, it does not load any data, just healpix pixels and other metadata
des_catalog = read_hats(DES_HATS_DIR)
plot_pixels(des_catalog)

[8]:

(<Figure size 1000x500 with 2 Axes>,
 <WCSAxes: title={'center': 'Catalog pixel map - des_dr2'}>)

../../_images/tutorials_pre_executed_des-gaia_18_1.png

This is what you would get for the full catalog, it would have many healpix tiles instead of just a single one. The reason is that hats-import splits the data into multiple files to have no more than a million rows per healpix tile / HATS parquet file.

DES Full Catalog Tile map

Compare it to the DES DR2 footprint:

4.3. Convert Gaia DR3 to HATS#

For Gaia we need to specify schema of the input data, because currently hats-import cannot infer column data types properly from this dataset.

See this GitHub issue for more details: astronomy-commons/hats-import#225

[9]:

gaia_file = next(GAIA_DIR.glob("*.csv.gz"))
empty_astropy_table = ascii.read(gaia_file, format="ecsv", data_end=1)
empty_astropy_table.write(
    GAIA_SCHEMA_FILE,
    # Uncomment to overwrite existing schema file
    # overwrite=True,
)

[10]:

gaia_args = ImportArguments(
    # sort columns are optional and works only if few objects are very close to each other
    sort_columns="source_id",
    ra_column="ra",
    dec_column="dec",
    # We select all Gaia ECSV files skipping the schema file
    input_file_list=list(GAIA_DIR.glob("*.csv.gz")),
    file_reader=CsvReader(
        comment="#",
        schema_file=GAIA_SCHEMA_FILE,
        compression="gzip",
    ),
    use_schema_file=GAIA_SCHEMA_FILE,
    output_artifact_name=GAIA_HATS_NAME,
    output_path=HATS_DIR,
)

pipeline_with_client(gaia_args, client)

4.4. Plot the Gaia DR3 HATS catalog pixels#

[11]:

# It should be full sky for the whole catalog
gaia_catalog = read_hats(GAIA_HATS_DIR)
plot_pixels(gaia_catalog)

[11]:

(<Figure size 1000x500 with 2 Axes>,
 <WCSAxes: title={'center': 'Catalog pixel map - gaia_dr3'}>)

../../_images/tutorials_pre_executed_des-gaia_25_1.png

And this is how it would look like for the entire Gaia catalog. Notice how Milky Way region is denser and thus has higher healpix order (smaller tiles), since hats-import limits the maximum number of objects per pixel to be 1 million.

Entire Gaia Sky map

4.5. Generate margin cache for the Gaia HATS catalog#

LSDB requires right-catalog margin cache to generate the complete cross-match result. Without the margin cache, the objects located near the edges of Healpix tiles may be missed in the cross-match.

[12]:

margin_cache_args = MarginCacheArguments(
    input_catalog_path=GAIA_HATS_DIR,
    output_path=HATS_DIR,
    margin_threshold=5.0,  # arcsec
    output_artifact_name=GAIA_MARGIN_CACHE_NAME,
)

pipeline_with_client(margin_cache_args, client)

5. Cross-match the catalogs#

The cross-matching in LSDB is not symmetric, meaning the choice of which catalog is “left” and which is “right” is crucial. In our case, we cross-match DES (left) to Gaia (right). This setup generally allows for multiple DES objects to be matched to a single Gaia object, a result of margin caches. Margin caches are designed to prevent missing objects near the edges of Healpix tiles. However, they can lead to multiple matches, where the same Gaia object might match to one DES object in its own tile and another DES object in the neighboring tile that includes this Gaia object in its margin cache.

However, we do not expect it to be a frequent case, because the DES catalog is much deeper than Gaia, and we use a small cross-match radius.

Plan the pipeline with LSDB
Run the pipeline with Dask

[13]:

des_catalog = lsdb.open_catalog(DES_HATS_DIR)

# gaia_margin_cache_catalog = lsdb.open_catalog(GAIA_MARGIN_CACHE_DIR)
gaia_catalog = lsdb.open_catalog(GAIA_HATS_DIR)

xmatched = des_catalog.crossmatch(
    gaia_catalog,
    # Up to 1 arcsec distance, it is the default
    radius_arcsec=1.0,
    # Single closest object, it is the default
    n_neighbors=1,
    # Default would be to use names of the HATS catalogs
    suffixes=("_des", "_gaia"),
)

display(des_catalog)
display(gaia_catalog)
display(xmatched)

/astro/users/kmalanch/.virtualenvs/default/lib/python3.10/site-packages/lsdb/dask/crossmatch_catalog_data.py:147: RuntimeWarning: Right catalog does not have a margin cache. Results may be incomplete and/or inaccurate.
  warnings.warn(

lsdb Catalog des_dr2:

	COADD_OBJECT_ID	TILENAME	HPIX_32	HPIX_64	HPIX_1024	HPIX_4096	HPIX_16384	RA	DEC	ALPHAWIN_J2000	DELTAWIN_J2000	GALACTIC_L	GALACTIC_B	XWIN_IMAGE	YWIN_IMAGE	A_IMAGE	ERRA_IMAGE	B_IMAGE	ERRB_IMAGE	THETA_J2000	ERRTHETA_IMAGE	KRON_RADIUS	EBV_SFD98	MAG_AUTO_G_DERED	MAG_AUTO_R_DERED	MAG_AUTO_I_DERED	MAG_AUTO_Z_DERED	MAG_AUTO_Y_DERED	WAVG_MAG_PSF_G_DERED	WAVG_MAG_PSF_R_DERED	WAVG_MAG_PSF_I_DERED	WAVG_MAG_PSF_Z_DERED	WAVG_MAG_PSF_Y_DERED	EXTENDED_CLASS_COADD	EXTENDED_CLASS_WAVG	FLAGS_G	IMAFLAGS_ISO_G	NEPOCHS_G	FLAGS_R	IMAFLAGS_ISO_R	NEPOCHS_R	FLAGS_I	IMAFLAGS_ISO_I	NEPOCHS_I	FLAGS_Z	IMAFLAGS_ISO_Z	NEPOCHS_Z	FLAGS_Y	IMAFLAGS_ISO_Y	NEPOCHS_Y	XWIN_IMAGE_G	XWIN_IMAGE_R	XWIN_IMAGE_I	XWIN_IMAGE_Z	XWIN_IMAGE_Y	YWIN_IMAGE_G	YWIN_IMAGE_R	YWIN_IMAGE_I	YWIN_IMAGE_Z	YWIN_IMAGE_Y	X2WIN_IMAGE_G	X2WIN_IMAGE_R	X2WIN_IMAGE_I	X2WIN_IMAGE_Z	X2WIN_IMAGE_Y	Y2WIN_IMAGE_G	Y2WIN_IMAGE_R	Y2WIN_IMAGE_I	Y2WIN_IMAGE_Z	Y2WIN_IMAGE_Y	XYWIN_IMAGE_G	XYWIN_IMAGE_R	XYWIN_IMAGE_I	XYWIN_IMAGE_Z	XYWIN_IMAGE_Y	ERRX2WIN_IMAGE_G	ERRX2WIN_IMAGE_R	ERRX2WIN_IMAGE_I	ERRX2WIN_IMAGE_Z	ERRX2WIN_IMAGE_Y	ERRY2WIN_IMAGE_G	ERRY2WIN_IMAGE_R	ERRY2WIN_IMAGE_I	ERRY2WIN_IMAGE_Z	ERRY2WIN_IMAGE_Y	ERRXYWIN_IMAGE_G	ERRXYWIN_IMAGE_R	ERRXYWIN_IMAGE_I	ERRXYWIN_IMAGE_Z	ERRXYWIN_IMAGE_Y	AWIN_IMAGE_G	AWIN_IMAGE_R	AWIN_IMAGE_I	AWIN_IMAGE_Z	AWIN_IMAGE_Y	BWIN_IMAGE_G	BWIN_IMAGE_R	BWIN_IMAGE_I	BWIN_IMAGE_Z	BWIN_IMAGE_Y	THETAWIN_IMAGE_G	THETAWIN_IMAGE_R	THETAWIN_IMAGE_I	THETAWIN_IMAGE_Z	THETAWIN_IMAGE_Y	ERRAWIN_IMAGE_G	ERRAWIN_IMAGE_R	ERRAWIN_IMAGE_I	ERRAWIN_IMAGE_Z	ERRAWIN_IMAGE_Y	ERRBWIN_IMAGE_G	ERRBWIN_IMAGE_R	ERRBWIN_IMAGE_I	ERRBWIN_IMAGE_Z	ERRBWIN_IMAGE_Y	ERRTHETAWIN_IMAGE_G	ERRTHETAWIN_IMAGE_R	ERRTHETAWIN_IMAGE_I	ERRTHETAWIN_IMAGE_Z	ERRTHETAWIN_IMAGE_Y	FLUX_RADIUS_G	FLUX_RADIUS_R	FLUX_RADIUS_I	FLUX_RADIUS_Z	FLUX_RADIUS_Y	FWHM_IMAGE_G	FWHM_IMAGE_R	FWHM_IMAGE_I	FWHM_IMAGE_Z	FWHM_IMAGE_Y	ISOAREA_IMAGE_G	ISOAREA_IMAGE_R	ISOAREA_IMAGE_I	ISOAREA_IMAGE_Z	ISOAREA_IMAGE_Y	BACKGROUND_G	BACKGROUND_R	BACKGROUND_I	BACKGROUND_Z	BACKGROUND_Y	NITER_MODEL_G	NITER_MODEL_R	NITER_MODEL_I	NITER_MODEL_Z	NITER_MODEL_Y	KRON_RADIUS_G	KRON_RADIUS_R	KRON_RADIUS_I	KRON_RADIUS_Z	KRON_RADIUS_Y	MAG_AUTO_G	MAG_AUTO_R	MAG_AUTO_I	MAG_AUTO_Z	MAG_AUTO_Y	MAGERR_AUTO_G	MAGERR_AUTO_R	MAGERR_AUTO_I	MAGERR_AUTO_Z	MAGERR_AUTO_Y	WAVG_MAG_PSF_G	WAVG_MAG_PSF_R	WAVG_MAG_PSF_I	WAVG_MAG_PSF_Z	WAVG_MAG_PSF_Y	WAVG_MAGERR_PSF_G	WAVG_MAGERR_PSF_R	WAVG_MAGERR_PSF_I	WAVG_MAGERR_PSF_Z	WAVG_MAGERR_PSF_Y	FLUX_AUTO_G	FLUX_AUTO_R	FLUX_AUTO_I	FLUX_AUTO_Z	FLUX_AUTO_Y	FLUXERR_AUTO_G	FLUXERR_AUTO_R	FLUXERR_AUTO_I	FLUXERR_AUTO_Z	FLUXERR_AUTO_Y	WAVG_FLUX_PSF_G	WAVG_FLUX_PSF_R	WAVG_FLUX_PSF_I	WAVG_FLUX_PSF_Z	WAVG_FLUX_PSF_Y	WAVG_FLUXERR_PSF_G	WAVG_FLUXERR_PSF_R	WAVG_FLUXERR_PSF_I	WAVG_FLUXERR_PSF_Z	WAVG_FLUXERR_PSF_Y	CLASS_STAR_G	CLASS_STAR_R	CLASS_STAR_I	CLASS_STAR_Z	CLASS_STAR_Y	SPREAD_MODEL_G	SPREAD_MODEL_R	SPREAD_MODEL_I	SPREAD_MODEL_Z	SPREAD_MODEL_Y	WAVG_SPREAD_MODEL_G	WAVG_SPREAD_MODEL_R	WAVG_SPREAD_MODEL_I	WAVG_SPREAD_MODEL_Z	WAVG_SPREAD_MODEL_Y	SPREADERR_MODEL_G	SPREADERR_MODEL_R	SPREADERR_MODEL_I	SPREADERR_MODEL_Z	SPREADERR_MODEL_Y	WAVG_SPREADERR_MODEL_G	WAVG_SPREADERR_MODEL_R	WAVG_SPREADERR_MODEL_I	WAVG_SPREADERR_MODEL_Z	WAVG_SPREADERR_MODEL_Y
npartitions=1
Order: 4, Pixel: 1216	int64[pyarrow]	string[pyarrow]	int64[pyarrow]	int64[pyarrow]	int64[pyarrow]	int64[pyarrow]	int64[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]

215 out of 215 columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

lsdb Catalog gaia_dr3:

	solution_id	designation	source_id	random_index	ref_epoch	ra	ra_error	dec	dec_error	parallax	parallax_error	parallax_over_error	pm	pmra	pmra_error	pmdec	pmdec_error	ra_dec_corr	ra_parallax_corr	ra_pmra_corr	ra_pmdec_corr	dec_parallax_corr	dec_pmra_corr	dec_pmdec_corr	parallax_pmra_corr	parallax_pmdec_corr	pmra_pmdec_corr	astrometric_n_obs_al	astrometric_n_obs_ac	astrometric_n_good_obs_al	astrometric_n_bad_obs_al	astrometric_gof_al	astrometric_chi2_al	astrometric_excess_noise	astrometric_excess_noise_sig	astrometric_params_solved	astrometric_primary_flag	nu_eff_used_in_astrometry	pseudocolour	pseudocolour_error	ra_pseudocolour_corr	dec_pseudocolour_corr	parallax_pseudocolour_corr	pmra_pseudocolour_corr	pmdec_pseudocolour_corr	astrometric_matched_transits	visibility_periods_used	astrometric_sigma5d_max	matched_transits	new_matched_transits	matched_transits_removed	ipd_gof_harmonic_amplitude	ipd_gof_harmonic_phase	ipd_frac_multi_peak	ipd_frac_odd_win	ruwe	scan_direction_strength_k1	scan_direction_strength_k2	scan_direction_strength_k3	scan_direction_strength_k4	scan_direction_mean_k1	scan_direction_mean_k2	scan_direction_mean_k3	scan_direction_mean_k4	duplicated_source	phot_g_n_obs	phot_g_mean_flux	phot_g_mean_flux_error	phot_g_mean_flux_over_error	phot_g_mean_mag	phot_bp_n_obs	phot_bp_mean_flux	phot_bp_mean_flux_error	phot_bp_mean_flux_over_error	phot_bp_mean_mag	phot_rp_n_obs	phot_rp_mean_flux	phot_rp_mean_flux_error	phot_rp_mean_flux_over_error	phot_rp_mean_mag	phot_bp_rp_excess_factor	phot_bp_n_contaminated_transits	phot_bp_n_blended_transits	phot_rp_n_contaminated_transits	phot_rp_n_blended_transits	phot_proc_mode	bp_rp	bp_g	g_rp	radial_velocity	radial_velocity_error	rv_method_used	rv_nb_transits	rv_nb_deblended_transits	rv_visibility_periods_used	rv_expected_sig_to_noise	rv_renormalised_gof	rv_chisq_pvalue	rv_time_duration	rv_amplitude_robust	rv_template_teff	rv_template_logg	rv_template_fe_h	rv_atm_param_origin	vbroad	vbroad_error	vbroad_nb_transits	grvs_mag	grvs_mag_error	grvs_mag_nb_transits	rvs_spec_sig_to_noise	phot_variable_flag	l	b	ecl_lon	ecl_lat	in_qso_candidates	in_galaxy_candidates	non_single_star	has_xp_continuous	has_xp_sampled	has_rvs	has_epoch_photometry	has_epoch_rv	has_mcmc_gspphot	has_mcmc_msc	in_andromeda_survey	classprob_dsc_combmod_quasar	classprob_dsc_combmod_galaxy	classprob_dsc_combmod_star	teff_gspphot	teff_gspphot_lower	teff_gspphot_upper	logg_gspphot	logg_gspphot_lower	logg_gspphot_upper	mh_gspphot	mh_gspphot_lower	mh_gspphot_upper	distance_gspphot	distance_gspphot_lower	distance_gspphot_upper	azero_gspphot	azero_gspphot_lower	azero_gspphot_upper	ag_gspphot	ag_gspphot_lower	ag_gspphot_upper	ebpminrp_gspphot	ebpminrp_gspphot_lower	ebpminrp_gspphot_upper	libname_gspphot
npartitions=1
Order: 0, Pixel: 4	int64[pyarrow]	string[pyarrow]	int64[pyarrow]	int64[pyarrow]	double[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int8[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	int8[pyarrow]	int8[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	bool[pyarrow]	int16[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int8[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int8[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	float[pyarrow]	string[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	bool[pyarrow]	bool[pyarrow]	int16[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	string[pyarrow]

152 out of 152 columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

lsdb Catalog des_dr2_x_gaia_dr3:

	COADD_OBJECT_ID_des	TILENAME_des	HPIX_32_des	HPIX_64_des	HPIX_1024_des	HPIX_4096_des	HPIX_16384_des	RA_des	DEC_des	ALPHAWIN_J2000_des	DELTAWIN_J2000_des	GALACTIC_L_des	GALACTIC_B_des	XWIN_IMAGE_des	YWIN_IMAGE_des	A_IMAGE_des	ERRA_IMAGE_des	B_IMAGE_des	ERRB_IMAGE_des	THETA_J2000_des	ERRTHETA_IMAGE_des	KRON_RADIUS_des	EBV_SFD98_des	MAG_AUTO_G_DERED_des	MAG_AUTO_R_DERED_des	MAG_AUTO_I_DERED_des	MAG_AUTO_Z_DERED_des	MAG_AUTO_Y_DERED_des	WAVG_MAG_PSF_G_DERED_des	WAVG_MAG_PSF_R_DERED_des	WAVG_MAG_PSF_I_DERED_des	WAVG_MAG_PSF_Z_DERED_des	WAVG_MAG_PSF_Y_DERED_des	EXTENDED_CLASS_COADD_des	EXTENDED_CLASS_WAVG_des	FLAGS_G_des	IMAFLAGS_ISO_G_des	NEPOCHS_G_des	FLAGS_R_des	IMAFLAGS_ISO_R_des	NEPOCHS_R_des	FLAGS_I_des	IMAFLAGS_ISO_I_des	NEPOCHS_I_des	FLAGS_Z_des	IMAFLAGS_ISO_Z_des	NEPOCHS_Z_des	FLAGS_Y_des	IMAFLAGS_ISO_Y_des	NEPOCHS_Y_des	XWIN_IMAGE_G_des	XWIN_IMAGE_R_des	XWIN_IMAGE_I_des	XWIN_IMAGE_Z_des	XWIN_IMAGE_Y_des	YWIN_IMAGE_G_des	YWIN_IMAGE_R_des	YWIN_IMAGE_I_des	YWIN_IMAGE_Z_des	YWIN_IMAGE_Y_des	X2WIN_IMAGE_G_des	X2WIN_IMAGE_R_des	X2WIN_IMAGE_I_des	X2WIN_IMAGE_Z_des	X2WIN_IMAGE_Y_des	Y2WIN_IMAGE_G_des	Y2WIN_IMAGE_R_des	Y2WIN_IMAGE_I_des	Y2WIN_IMAGE_Z_des	Y2WIN_IMAGE_Y_des	XYWIN_IMAGE_G_des	XYWIN_IMAGE_R_des	XYWIN_IMAGE_I_des	XYWIN_IMAGE_Z_des	XYWIN_IMAGE_Y_des	ERRX2WIN_IMAGE_G_des	ERRX2WIN_IMAGE_R_des	ERRX2WIN_IMAGE_I_des	ERRX2WIN_IMAGE_Z_des	ERRX2WIN_IMAGE_Y_des	ERRY2WIN_IMAGE_G_des	ERRY2WIN_IMAGE_R_des	ERRY2WIN_IMAGE_I_des	ERRY2WIN_IMAGE_Z_des	ERRY2WIN_IMAGE_Y_des	ERRXYWIN_IMAGE_G_des	ERRXYWIN_IMAGE_R_des	ERRXYWIN_IMAGE_I_des	ERRXYWIN_IMAGE_Z_des	ERRXYWIN_IMAGE_Y_des	AWIN_IMAGE_G_des	AWIN_IMAGE_R_des	AWIN_IMAGE_I_des	AWIN_IMAGE_Z_des	AWIN_IMAGE_Y_des	BWIN_IMAGE_G_des	BWIN_IMAGE_R_des	BWIN_IMAGE_I_des	BWIN_IMAGE_Z_des	BWIN_IMAGE_Y_des	THETAWIN_IMAGE_G_des	THETAWIN_IMAGE_R_des	THETAWIN_IMAGE_I_des	THETAWIN_IMAGE_Z_des	THETAWIN_IMAGE_Y_des	ERRAWIN_IMAGE_G_des	ERRAWIN_IMAGE_R_des	ERRAWIN_IMAGE_I_des	ERRAWIN_IMAGE_Z_des	ERRAWIN_IMAGE_Y_des	ERRBWIN_IMAGE_G_des	ERRBWIN_IMAGE_R_des	ERRBWIN_IMAGE_I_des	ERRBWIN_IMAGE_Z_des	ERRBWIN_IMAGE_Y_des	ERRTHETAWIN_IMAGE_G_des	ERRTHETAWIN_IMAGE_R_des	ERRTHETAWIN_IMAGE_I_des	ERRTHETAWIN_IMAGE_Z_des	ERRTHETAWIN_IMAGE_Y_des	FLUX_RADIUS_G_des	FLUX_RADIUS_R_des	FLUX_RADIUS_I_des	FLUX_RADIUS_Z_des	FLUX_RADIUS_Y_des	FWHM_IMAGE_G_des	FWHM_IMAGE_R_des	FWHM_IMAGE_I_des	FWHM_IMAGE_Z_des	FWHM_IMAGE_Y_des	ISOAREA_IMAGE_G_des	ISOAREA_IMAGE_R_des	ISOAREA_IMAGE_I_des	ISOAREA_IMAGE_Z_des	ISOAREA_IMAGE_Y_des	BACKGROUND_G_des	BACKGROUND_R_des	BACKGROUND_I_des	BACKGROUND_Z_des	BACKGROUND_Y_des	NITER_MODEL_G_des	NITER_MODEL_R_des	NITER_MODEL_I_des	NITER_MODEL_Z_des	NITER_MODEL_Y_des	KRON_RADIUS_G_des	KRON_RADIUS_R_des	KRON_RADIUS_I_des	KRON_RADIUS_Z_des	KRON_RADIUS_Y_des	MAG_AUTO_G_des	MAG_AUTO_R_des	MAG_AUTO_I_des	MAG_AUTO_Z_des	MAG_AUTO_Y_des	MAGERR_AUTO_G_des	MAGERR_AUTO_R_des	MAGERR_AUTO_I_des	MAGERR_AUTO_Z_des	MAGERR_AUTO_Y_des	WAVG_MAG_PSF_G_des	WAVG_MAG_PSF_R_des	WAVG_MAG_PSF_I_des	WAVG_MAG_PSF_Z_des	WAVG_MAG_PSF_Y_des	WAVG_MAGERR_PSF_G_des	WAVG_MAGERR_PSF_R_des	WAVG_MAGERR_PSF_I_des	WAVG_MAGERR_PSF_Z_des	WAVG_MAGERR_PSF_Y_des	FLUX_AUTO_G_des	FLUX_AUTO_R_des	FLUX_AUTO_I_des	FLUX_AUTO_Z_des	FLUX_AUTO_Y_des	FLUXERR_AUTO_G_des	FLUXERR_AUTO_R_des	FLUXERR_AUTO_I_des	FLUXERR_AUTO_Z_des	FLUXERR_AUTO_Y_des	WAVG_FLUX_PSF_G_des	WAVG_FLUX_PSF_R_des	WAVG_FLUX_PSF_I_des	WAVG_FLUX_PSF_Z_des	WAVG_FLUX_PSF_Y_des	WAVG_FLUXERR_PSF_G_des	WAVG_FLUXERR_PSF_R_des	WAVG_FLUXERR_PSF_I_des	WAVG_FLUXERR_PSF_Z_des	WAVG_FLUXERR_PSF_Y_des	CLASS_STAR_G_des	CLASS_STAR_R_des	CLASS_STAR_I_des	CLASS_STAR_Z_des	CLASS_STAR_Y_des	SPREAD_MODEL_G_des	SPREAD_MODEL_R_des	SPREAD_MODEL_I_des	SPREAD_MODEL_Z_des	SPREAD_MODEL_Y_des	WAVG_SPREAD_MODEL_G_des	WAVG_SPREAD_MODEL_R_des	WAVG_SPREAD_MODEL_I_des	WAVG_SPREAD_MODEL_Z_des	WAVG_SPREAD_MODEL_Y_des	SPREADERR_MODEL_G_des	SPREADERR_MODEL_R_des	SPREADERR_MODEL_I_des	SPREADERR_MODEL_Z_des	SPREADERR_MODEL_Y_des	WAVG_SPREADERR_MODEL_G_des	WAVG_SPREADERR_MODEL_R_des	WAVG_SPREADERR_MODEL_I_des	WAVG_SPREADERR_MODEL_Z_des	WAVG_SPREADERR_MODEL_Y_des	solution_id_gaia	designation_gaia	source_id_gaia	random_index_gaia	ref_epoch_gaia	ra_gaia	ra_error_gaia	dec_gaia	dec_error_gaia	parallax_gaia	parallax_error_gaia	parallax_over_error_gaia	pm_gaia	pmra_gaia	pmra_error_gaia	pmdec_gaia	pmdec_error_gaia	ra_dec_corr_gaia	ra_parallax_corr_gaia	ra_pmra_corr_gaia	ra_pmdec_corr_gaia	dec_parallax_corr_gaia	dec_pmra_corr_gaia	dec_pmdec_corr_gaia	parallax_pmra_corr_gaia	parallax_pmdec_corr_gaia	pmra_pmdec_corr_gaia	astrometric_n_obs_al_gaia	astrometric_n_obs_ac_gaia	astrometric_n_good_obs_al_gaia	astrometric_n_bad_obs_al_gaia	astrometric_gof_al_gaia	astrometric_chi2_al_gaia	astrometric_excess_noise_gaia	astrometric_excess_noise_sig_gaia	astrometric_params_solved_gaia	astrometric_primary_flag_gaia	nu_eff_used_in_astrometry_gaia	pseudocolour_gaia	pseudocolour_error_gaia	ra_pseudocolour_corr_gaia	dec_pseudocolour_corr_gaia	parallax_pseudocolour_corr_gaia	pmra_pseudocolour_corr_gaia	pmdec_pseudocolour_corr_gaia	astrometric_matched_transits_gaia	visibility_periods_used_gaia	astrometric_sigma5d_max_gaia	matched_transits_gaia	new_matched_transits_gaia	matched_transits_removed_gaia	ipd_gof_harmonic_amplitude_gaia	ipd_gof_harmonic_phase_gaia	ipd_frac_multi_peak_gaia	ipd_frac_odd_win_gaia	ruwe_gaia	scan_direction_strength_k1_gaia	scan_direction_strength_k2_gaia	scan_direction_strength_k3_gaia	scan_direction_strength_k4_gaia	scan_direction_mean_k1_gaia	scan_direction_mean_k2_gaia	scan_direction_mean_k3_gaia	scan_direction_mean_k4_gaia	duplicated_source_gaia	phot_g_n_obs_gaia	phot_g_mean_flux_gaia	phot_g_mean_flux_error_gaia	phot_g_mean_flux_over_error_gaia	phot_g_mean_mag_gaia	phot_bp_n_obs_gaia	phot_bp_mean_flux_gaia	phot_bp_mean_flux_error_gaia	phot_bp_mean_flux_over_error_gaia	phot_bp_mean_mag_gaia	phot_rp_n_obs_gaia	phot_rp_mean_flux_gaia	phot_rp_mean_flux_error_gaia	phot_rp_mean_flux_over_error_gaia	phot_rp_mean_mag_gaia	phot_bp_rp_excess_factor_gaia	phot_bp_n_contaminated_transits_gaia	phot_bp_n_blended_transits_gaia	phot_rp_n_contaminated_transits_gaia	phot_rp_n_blended_transits_gaia	phot_proc_mode_gaia	bp_rp_gaia	bp_g_gaia	g_rp_gaia	radial_velocity_gaia	radial_velocity_error_gaia	rv_method_used_gaia	rv_nb_transits_gaia	rv_nb_deblended_transits_gaia	rv_visibility_periods_used_gaia	rv_expected_sig_to_noise_gaia	rv_renormalised_gof_gaia	rv_chisq_pvalue_gaia	rv_time_duration_gaia	rv_amplitude_robust_gaia	rv_template_teff_gaia	rv_template_logg_gaia	rv_template_fe_h_gaia	rv_atm_param_origin_gaia	vbroad_gaia	vbroad_error_gaia	vbroad_nb_transits_gaia	grvs_mag_gaia	grvs_mag_error_gaia	grvs_mag_nb_transits_gaia	rvs_spec_sig_to_noise_gaia	phot_variable_flag_gaia	l_gaia	b_gaia	ecl_lon_gaia	ecl_lat_gaia	in_qso_candidates_gaia	in_galaxy_candidates_gaia	non_single_star_gaia	has_xp_continuous_gaia	has_xp_sampled_gaia	has_rvs_gaia	has_epoch_photometry_gaia	has_epoch_rv_gaia	has_mcmc_gspphot_gaia	has_mcmc_msc_gaia	in_andromeda_survey_gaia	classprob_dsc_combmod_quasar_gaia	classprob_dsc_combmod_galaxy_gaia	classprob_dsc_combmod_star_gaia	teff_gspphot_gaia	teff_gspphot_lower_gaia	teff_gspphot_upper_gaia	logg_gspphot_gaia	logg_gspphot_lower_gaia	logg_gspphot_upper_gaia	mh_gspphot_gaia	mh_gspphot_lower_gaia	mh_gspphot_upper_gaia	distance_gspphot_gaia	distance_gspphot_lower_gaia	distance_gspphot_upper_gaia	azero_gspphot_gaia	azero_gspphot_lower_gaia	azero_gspphot_upper_gaia	ag_gspphot_gaia	ag_gspphot_lower_gaia	ag_gspphot_upper_gaia	ebpminrp_gspphot_gaia	ebpminrp_gspphot_lower_gaia	ebpminrp_gspphot_upper_gaia	libname_gspphot_gaia	_dist_arcsec
npartitions=1
Order: 4, Pixel: 1216	int64[pyarrow]	string[pyarrow]	int64[pyarrow]	int64[pyarrow]	int64[pyarrow]	int64[pyarrow]	int64[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	int16[pyarrow]	int32[pyarrow]	int32[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	int32[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	int64[pyarrow]	string[pyarrow]	int64[pyarrow]	int64[pyarrow]	double[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int8[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	int8[pyarrow]	int8[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	bool[pyarrow]	int16[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	double[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	int8[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int8[pyarrow]	int16[pyarrow]	int16[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	float[pyarrow]	float[pyarrow]	int16[pyarrow]	float[pyarrow]	string[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	double[pyarrow]	bool[pyarrow]	bool[pyarrow]	int16[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	bool[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	float[pyarrow]	string[pyarrow]	double[pyarrow]

368 out of 368 columns in the catalog have been loaded lazily, meaning no data has been read, only the catalog schema

[ ]:

# Run the pipeline with Dask client, it will take a while
xmatched.write_catalog(XMATCH_DIR)

[15]:

# Look into the data
lsdb.open_catalog(XMATCH_DIR).head()

[15]:

	COADD_OBJECT_ID_des	TILENAME_des	HPIX_32_des	HPIX_64_des	HPIX_1024_des	HPIX_4096_des	HPIX_16384_des	RA_des	DEC_des	ALPHAWIN_J2000_des	DELTAWIN_J2000_des	GALACTIC_L_des	GALACTIC_B_des	XWIN_IMAGE_des	YWIN_IMAGE_des	A_IMAGE_des	ERRA_IMAGE_des	B_IMAGE_des	ERRB_IMAGE_des	THETA_J2000_des	ERRTHETA_IMAGE_des	KRON_RADIUS_des	EBV_SFD98_des	MAG_AUTO_G_DERED_des	MAG_AUTO_R_DERED_des	MAG_AUTO_I_DERED_des	MAG_AUTO_Z_DERED_des	MAG_AUTO_Y_DERED_des	WAVG_MAG_PSF_G_DERED_des	WAVG_MAG_PSF_R_DERED_des	WAVG_MAG_PSF_I_DERED_des	WAVG_MAG_PSF_Z_DERED_des	WAVG_MAG_PSF_Y_DERED_des	EXTENDED_CLASS_COADD_des	EXTENDED_CLASS_WAVG_des	FLAGS_G_des	IMAFLAGS_ISO_G_des	NEPOCHS_G_des	FLAGS_R_des	IMAFLAGS_ISO_R_des	NEPOCHS_R_des	FLAGS_I_des	IMAFLAGS_ISO_I_des	NEPOCHS_I_des	FLAGS_Z_des	IMAFLAGS_ISO_Z_des	NEPOCHS_Z_des	FLAGS_Y_des	IMAFLAGS_ISO_Y_des	NEPOCHS_Y_des	XWIN_IMAGE_G_des	XWIN_IMAGE_R_des	XWIN_IMAGE_I_des	XWIN_IMAGE_Z_des	XWIN_IMAGE_Y_des	YWIN_IMAGE_G_des	YWIN_IMAGE_R_des	YWIN_IMAGE_I_des	YWIN_IMAGE_Z_des	YWIN_IMAGE_Y_des	X2WIN_IMAGE_G_des	X2WIN_IMAGE_R_des	X2WIN_IMAGE_I_des	X2WIN_IMAGE_Z_des	X2WIN_IMAGE_Y_des	Y2WIN_IMAGE_G_des	Y2WIN_IMAGE_R_des	Y2WIN_IMAGE_I_des	Y2WIN_IMAGE_Z_des	Y2WIN_IMAGE_Y_des	XYWIN_IMAGE_G_des	XYWIN_IMAGE_R_des	XYWIN_IMAGE_I_des	XYWIN_IMAGE_Z_des	XYWIN_IMAGE_Y_des	ERRX2WIN_IMAGE_G_des	ERRX2WIN_IMAGE_R_des	ERRX2WIN_IMAGE_I_des	ERRX2WIN_IMAGE_Z_des	ERRX2WIN_IMAGE_Y_des	ERRY2WIN_IMAGE_G_des	ERRY2WIN_IMAGE_R_des	ERRY2WIN_IMAGE_I_des	ERRY2WIN_IMAGE_Z_des	ERRY2WIN_IMAGE_Y_des	ERRXYWIN_IMAGE_G_des	ERRXYWIN_IMAGE_R_des	ERRXYWIN_IMAGE_I_des	ERRXYWIN_IMAGE_Z_des	ERRXYWIN_IMAGE_Y_des	AWIN_IMAGE_G_des	AWIN_IMAGE_R_des	AWIN_IMAGE_I_des	AWIN_IMAGE_Z_des	AWIN_IMAGE_Y_des	BWIN_IMAGE_G_des	BWIN_IMAGE_R_des	BWIN_IMAGE_I_des	BWIN_IMAGE_Z_des	BWIN_IMAGE_Y_des	THETAWIN_IMAGE_G_des	THETAWIN_IMAGE_R_des	THETAWIN_IMAGE_I_des	THETAWIN_IMAGE_Z_des	THETAWIN_IMAGE_Y_des	ERRAWIN_IMAGE_G_des	ERRAWIN_IMAGE_R_des	ERRAWIN_IMAGE_I_des	ERRAWIN_IMAGE_Z_des	ERRAWIN_IMAGE_Y_des	ERRBWIN_IMAGE_G_des	ERRBWIN_IMAGE_R_des	ERRBWIN_IMAGE_I_des	ERRBWIN_IMAGE_Z_des	ERRBWIN_IMAGE_Y_des	ERRTHETAWIN_IMAGE_G_des	ERRTHETAWIN_IMAGE_R_des	ERRTHETAWIN_IMAGE_I_des	ERRTHETAWIN_IMAGE_Z_des	ERRTHETAWIN_IMAGE_Y_des	FLUX_RADIUS_G_des	FLUX_RADIUS_R_des	FLUX_RADIUS_I_des	FLUX_RADIUS_Z_des	FLUX_RADIUS_Y_des	FWHM_IMAGE_G_des	FWHM_IMAGE_R_des	FWHM_IMAGE_I_des	FWHM_IMAGE_Z_des	FWHM_IMAGE_Y_des	ISOAREA_IMAGE_G_des	ISOAREA_IMAGE_R_des	ISOAREA_IMAGE_I_des	ISOAREA_IMAGE_Z_des	ISOAREA_IMAGE_Y_des	BACKGROUND_G_des	BACKGROUND_R_des	BACKGROUND_I_des	BACKGROUND_Z_des	BACKGROUND_Y_des	NITER_MODEL_G_des	NITER_MODEL_R_des	NITER_MODEL_I_des	NITER_MODEL_Z_des	NITER_MODEL_Y_des	KRON_RADIUS_G_des	KRON_RADIUS_R_des	KRON_RADIUS_I_des	KRON_RADIUS_Z_des	KRON_RADIUS_Y_des	MAG_AUTO_G_des	MAG_AUTO_R_des	MAG_AUTO_I_des	MAG_AUTO_Z_des	MAG_AUTO_Y_des	MAGERR_AUTO_G_des	MAGERR_AUTO_R_des	MAGERR_AUTO_I_des	MAGERR_AUTO_Z_des	MAGERR_AUTO_Y_des	WAVG_MAG_PSF_G_des	WAVG_MAG_PSF_R_des	WAVG_MAG_PSF_I_des	WAVG_MAG_PSF_Z_des	WAVG_MAG_PSF_Y_des	WAVG_MAGERR_PSF_G_des	WAVG_MAGERR_PSF_R_des	WAVG_MAGERR_PSF_I_des	WAVG_MAGERR_PSF_Z_des	WAVG_MAGERR_PSF_Y_des	FLUX_AUTO_G_des	FLUX_AUTO_R_des	FLUX_AUTO_I_des	FLUX_AUTO_Z_des	FLUX_AUTO_Y_des	FLUXERR_AUTO_G_des	FLUXERR_AUTO_R_des	FLUXERR_AUTO_I_des	FLUXERR_AUTO_Z_des	FLUXERR_AUTO_Y_des	WAVG_FLUX_PSF_G_des	WAVG_FLUX_PSF_R_des	WAVG_FLUX_PSF_I_des	WAVG_FLUX_PSF_Z_des	WAVG_FLUX_PSF_Y_des	WAVG_FLUXERR_PSF_G_des	WAVG_FLUXERR_PSF_R_des	WAVG_FLUXERR_PSF_I_des	WAVG_FLUXERR_PSF_Z_des	WAVG_FLUXERR_PSF_Y_des	CLASS_STAR_G_des	CLASS_STAR_R_des	CLASS_STAR_I_des	CLASS_STAR_Z_des	CLASS_STAR_Y_des	SPREAD_MODEL_G_des	SPREAD_MODEL_R_des	SPREAD_MODEL_I_des	SPREAD_MODEL_Z_des	SPREAD_MODEL_Y_des	WAVG_SPREAD_MODEL_G_des	WAVG_SPREAD_MODEL_R_des	WAVG_SPREAD_MODEL_I_des	WAVG_SPREAD_MODEL_Z_des	WAVG_SPREAD_MODEL_Y_des	SPREADERR_MODEL_G_des	SPREADERR_MODEL_R_des	SPREADERR_MODEL_I_des	SPREADERR_MODEL_Z_des	SPREADERR_MODEL_Y_des	WAVG_SPREADERR_MODEL_G_des	WAVG_SPREADERR_MODEL_R_des	WAVG_SPREADERR_MODEL_I_des	WAVG_SPREADERR_MODEL_Z_des	WAVG_SPREADERR_MODEL_Y_des	solution_id_gaia	designation_gaia	source_id_gaia	random_index_gaia	ref_epoch_gaia	ra_gaia	ra_error_gaia	dec_gaia	dec_error_gaia	parallax_gaia	parallax_error_gaia	parallax_over_error_gaia	pm_gaia	pmra_gaia	pmra_error_gaia	pmdec_gaia	pmdec_error_gaia	ra_dec_corr_gaia	ra_parallax_corr_gaia	ra_pmra_corr_gaia	ra_pmdec_corr_gaia	dec_parallax_corr_gaia	dec_pmra_corr_gaia	dec_pmdec_corr_gaia	parallax_pmra_corr_gaia	parallax_pmdec_corr_gaia	pmra_pmdec_corr_gaia	astrometric_n_obs_al_gaia	astrometric_n_obs_ac_gaia	astrometric_n_good_obs_al_gaia	astrometric_n_bad_obs_al_gaia	astrometric_gof_al_gaia	astrometric_chi2_al_gaia	astrometric_excess_noise_gaia	astrometric_excess_noise_sig_gaia	astrometric_params_solved_gaia	astrometric_primary_flag_gaia	nu_eff_used_in_astrometry_gaia	pseudocolour_gaia	pseudocolour_error_gaia	ra_pseudocolour_corr_gaia	dec_pseudocolour_corr_gaia	parallax_pseudocolour_corr_gaia	pmra_pseudocolour_corr_gaia	pmdec_pseudocolour_corr_gaia	astrometric_matched_transits_gaia	visibility_periods_used_gaia	astrometric_sigma5d_max_gaia	matched_transits_gaia	new_matched_transits_gaia	matched_transits_removed_gaia	ipd_gof_harmonic_amplitude_gaia	ipd_gof_harmonic_phase_gaia	ipd_frac_multi_peak_gaia	ipd_frac_odd_win_gaia	ruwe_gaia	scan_direction_strength_k1_gaia	scan_direction_strength_k2_gaia	scan_direction_strength_k3_gaia	scan_direction_strength_k4_gaia	scan_direction_mean_k1_gaia	scan_direction_mean_k2_gaia	scan_direction_mean_k3_gaia	scan_direction_mean_k4_gaia	duplicated_source_gaia	phot_g_n_obs_gaia	phot_g_mean_flux_gaia	phot_g_mean_flux_error_gaia	phot_g_mean_flux_over_error_gaia	phot_g_mean_mag_gaia	phot_bp_n_obs_gaia	phot_bp_mean_flux_gaia	phot_bp_mean_flux_error_gaia	phot_bp_mean_flux_over_error_gaia	phot_bp_mean_mag_gaia	phot_rp_n_obs_gaia	phot_rp_mean_flux_gaia	phot_rp_mean_flux_error_gaia	phot_rp_mean_flux_over_error_gaia	phot_rp_mean_mag_gaia	phot_bp_rp_excess_factor_gaia	phot_bp_n_contaminated_transits_gaia	phot_bp_n_blended_transits_gaia	phot_rp_n_contaminated_transits_gaia	phot_rp_n_blended_transits_gaia	phot_proc_mode_gaia	bp_rp_gaia	bp_g_gaia	g_rp_gaia	radial_velocity_gaia	radial_velocity_error_gaia	rv_method_used_gaia	rv_nb_transits_gaia	rv_nb_deblended_transits_gaia	rv_visibility_periods_used_gaia	rv_expected_sig_to_noise_gaia	rv_renormalised_gof_gaia	rv_chisq_pvalue_gaia	rv_time_duration_gaia	rv_amplitude_robust_gaia	rv_template_teff_gaia	rv_template_logg_gaia	rv_template_fe_h_gaia	rv_atm_param_origin_gaia	vbroad_gaia	vbroad_error_gaia	vbroad_nb_transits_gaia	grvs_mag_gaia	grvs_mag_error_gaia	grvs_mag_nb_transits_gaia	rvs_spec_sig_to_noise_gaia	phot_variable_flag_gaia	l_gaia	b_gaia	ecl_lon_gaia	ecl_lat_gaia	in_qso_candidates_gaia	in_galaxy_candidates_gaia	non_single_star_gaia	has_xp_continuous_gaia	has_xp_sampled_gaia	has_rvs_gaia	has_epoch_photometry_gaia	has_epoch_rv_gaia	has_mcmc_gspphot_gaia	has_mcmc_msc_gaia	in_andromeda_survey_gaia	classprob_dsc_combmod_quasar_gaia	classprob_dsc_combmod_galaxy_gaia	classprob_dsc_combmod_star_gaia	teff_gspphot_gaia	teff_gspphot_lower_gaia	teff_gspphot_upper_gaia	logg_gspphot_gaia	logg_gspphot_lower_gaia	logg_gspphot_upper_gaia	mh_gspphot_gaia	mh_gspphot_lower_gaia	mh_gspphot_upper_gaia	distance_gspphot_gaia	distance_gspphot_lower_gaia	distance_gspphot_upper_gaia	azero_gspphot_gaia	azero_gspphot_lower_gaia	azero_gspphot_upper_gaia	ag_gspphot_gaia	ag_gspphot_lower_gaia	ag_gspphot_upper_gaia	ebpminrp_gspphot_gaia	ebpminrp_gspphot_lower_gaia	ebpminrp_gspphot_upper_gaia	libname_gspphot_gaia	_dist_arcsec
_healpix_29
1369333388684088685	1033365960	DES0000+0209	4864	19459	4981605	79705693	1275291097	0.315738	1.807978	0.315738	1.807978	98.277855	-58.641258	859.722573	205.047007	13.109224	0.021113	11.895746	0.019448	-8.348129	79.776337	3.5	0.02106	16.207703	16.14784	16.103861	15.802624	14.405433	-99.0	-99.0	-99.0	-99.0	-99.0	3	-9	2	1	0	2	1	0	2	1	0	2	1	0	2	1	0	859.929702	859.70388	859.862453	859.66912	862.564673	205.587072	205.070812	204.917383	205.065974	205.337321	31.071758	32.900852	38.409364	36.744125	20.215549	35.067481	44.706109	51.312286	52.420729	23.321088	0.134692	0.117458	0.11162	0.017416	0.072723	0.000061	0.000059	0.000107	0.00023	0.000032	0.000025	0.000022	0.00005	0.000116	0.00003	0.0	0.000001	0.000001	-0.000004	0.0	5.922163	6.68635	7.163327	7.240217	4.829368	5.573798	5.735825	6.197451	6.061692	4.495981	88.071526	89.430008	89.504402	89.936348	88.659264	0.007835	0.00765	0.010353	0.015181	0.005685	0.004985	0.004695	0.007093	0.01076	0.005492	0.315351	1.415489	0.946607	-1.970798	8.841293	9.620294	10.573933	11.34127	11.494138	8.83498	35.450596	23.626568	25.30064	24.08963	8.97525	6169	6793	6153	6151	6293	0.014667	0.016256	-0.07027	-0.163805	0.219037	99	47	71	43	95	3.5	3.5	3.5	3.5	3.5	16.274799	16.192909	16.136904	15.827811	14.427504	0.000779	0.000812	0.00123	0.001746	0.001409	-99.0	-99.0	-99.0	-99.0	-99.0	-99.0	-99.0	-99.0	-99.0	-99.0	309086.84375	333300.46875	350944.25	466525.75	1694332.125	221.589661	249.337418	397.384674	750.113525	2199.000977	inf	inf	inf	inf	inf	-inf	-inf	-inf	-inf	-inf	0.746572	0.835734	0.834762	0.822213	0.028636	0.040675	0.040326	0.043034	0.041587	0.022356	-99.0	-99.0	-99.0	-99.0	-99.0	0.000086	0.000062	0.000107	0.00015	0.000052	-99.0	-99.0	-99.0	-99.0	-99.0	1636148068921376768	Gaia DR3 2738666762216622080	2738666762216622080	1239508586	2016.0	0.315703	0.015816	1.808027	0.011115	4.143008	0.018335	225.96019	97.6782	35.879285	0.021572	-90.849921	0.011853	0.157948	0.236682	0.130806	-0.175823	-0.113477	-0.198022	0.59848	-0.060791	-0.34467	0.233159	353	0	352	1	0.269435	381.42218	0.016398	0.272375	31	False	1.517572	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	40	16	0.03028	48	7	0	0.033208	47.009277	0	0	1.009281	0.431008	0.649729	0.326442	0.251903	-24.759705	-16.164049	-20.843275	-3.343007	False	398	107777.39056	15.58101	6917.227539	13.106048	44	53535.174176	29.309889	1826.522583	13.516944	42	77659.992091	35.747173	2172.479248	12.522402	1.217279	0	9	0	7	0	0.994542	0.410896	0.583646	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	NOT_AVAILABLE	98.277826	-58.641201	1.009006	1.533218	False	False	0	False	False	False	False	False	True	True	False	0.0	0.0	0.999798	5313.100098	5304.98291	5320.712402	4.6864	4.6809	4.6961	-0.8667	-0.8822	-0.8543	231.736404	229.949097	233.156204	0.1894	0.1834	0.1946	0.1551	0.1502	0.1595	0.0839	0.0812	0.0863	MARCS	0.217639
1369333392965728211	1033367823	DES0000+0209	4864	19459	4981605	79705693	1275291101	0.321042	1.812476	0.321042	1.812476	98.29052	-58.639053	787.164865	266.638258	3.223952	0.00431	3.175361	0.004191	2.895157	-29.177334	3.5	0.02123	16.06711	15.761197	16.115875	15.892041	15.719292	16.067291	15.760271	15.686219	15.669059	15.703516	1	0	0	0	7	4	0	7	0	0	5	0	0	6	0	0	5	787.109436	787.143862	787.207075	787.156398	787.108863	266.569632	266.63489	266.626268	266.649556	266.680372	2.080884	1.518296	1.555147	1.269225	1.541184	2.130177	1.513978	1.527102	1.257572	1.547461	0.00367	-0.005985	-0.015427	-0.016557	-0.041231	0.0	0.0	0.0	0.0	0.000002	0.000001	0.0	0.0	0.0	0.000002	0.0	-0.0	-0.0	-0.0	-0.0	1.459606	1.233896	1.249789	1.131791	1.259235	1.442433	1.228729	1.232995	1.116175	1.225958	85.765335	-35.079723	-23.865425	-35.306522	-47.17654	0.000707	0.000491	0.000637	0.000622	0.001563	0.000701	0.000489	0.000629	0.000615	0.001533	89.512001	-37.025681	-16.162096	-30.132202	-20.157438	2.542885	2.165329	2.205929	2.008369	2.242565	4.217923	3.604914	3.516787	3.187831	3.407361	1545	1496	1321	1235	1085	0.0042	-0.005586	-0.072878	-0.149149	0.154202	56	55	94	49	55	3.5	3.5	3.5	3.5	3.5	16.134748	15.806629	16.149185	15.917432	15.741541	0.000286	0.000244	0.000373	0.000483	0.001267	16.13493	15.805703	15.719529	15.694449	15.725765	0.000358	0.000282	0.000344	0.000383	0.000931	351642.125	475716.5625	346997.46875	429563.3125	505107.09375	92.526093	107.002953	119.138527	191.159119	589.341797	351583.21875	476122.4375	515452.03125	527497.5	512500.03125	115.881248	123.862083	163.173767	185.902771	439.697113	0.999723	0.99924	0.997452	0.999243	0.859685	-0.000551	-0.001305	0.002612	0.00079	-0.004642	0.000002	-0.000074	-0.000162	-0.000102	-0.000207	0.000011	0.000009	0.000013	0.000013	0.000035	0.000014	0.000011	0.000013	0.000016	0.000039	1636148068921376768	Gaia DR3 2738666762216621952	2738666762216621952	9635706	2016.0	0.321041	0.044902	1.812475	0.028223	0.3589	0.050493	7.107963	24.915899	23.151403	0.060956	-9.209482	0.030257	0.136676	0.278616	0.044427	-0.24014	-0.121248	-0.247631	0.539106	0.023801	-0.345601	0.199123	355	0	354	1	1.611479	410.636536	0.056925	0.410084	31	True	1.579196	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	40	16	0.085018	48	7	0	0.036529	48.443333	0	0	1.060637	0.432468	0.650096	0.323673	0.283436	-23.164856	-15.084967	-19.023005	-2.091124	False	403	8857.814761	3.144313	2817.091064	15.819051	45	4947.481607	18.386446	269.083099	16.102581	41	5622.415177	11.604496	484.503174	15.373088	1.193285	0	1	0	2	0	0.729493	0.28353	0.445963	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	NOT_AVAILABLE	98.29052	-58.639052	1.015673	1.535176	False	False	0	False	False	False	False	False	True	True	False	0.0	0.0	0.999957	6098.600098	6053.160645	6138.477539	3.9914	3.9462	4.0423	-1.095	-1.148	-1.0456	2633.114502	2483.048584	2780.737549	0.1467	0.1189	0.1678	0.1258	0.1019	0.1442	0.0686	0.0556	0.0787	A	0.003322
1369333418327952083	1033367955	DES0000+0209	4864	19459	4981605	79705695	1275291124	0.310521	1.816329	0.310521	1.816329	98.274513	-58.631569	931.118713	319.360443	2.806581	0.015019	2.478078	0.013693	9.14357	-81.657364	3.5	0.02137	19.482012	19.098812	18.992083	18.966431	18.977551	19.48682	19.125164	19.02437	18.995228	19.02799	0	0	0	0	8	0	0	7	0	0	7	0	0	7	0	0	6	931.107668	931.122949	931.122061	931.111059	931.107481	319.311441	319.347988	319.377269	319.355969	319.393187	2.143901	1.746998	1.392521	1.201875	1.748469	2.232112	1.777809	1.413371	1.218535	1.826036	-0.009031	-0.008781	-0.023032	-0.013342	-0.076868	0.000023	0.000013	0.000014	0.000032	0.000614	0.000023	0.000013	0.000014	0.000031	0.000615	-0.0	-0.0	-0.0	-0.0	-0.000001	1.494332	1.334218	1.195085	1.107219	1.368704	1.463894	1.32086	1.173739	1.092921	1.304283	-84.214294	-75.158592	-57.176498	-60.988914	-58.386475	0.004792	0.003596	0.003739	0.005616	0.024818	0.004764	0.003588	0.003687	0.005609	0.024771	-85.159889	-84.82753	-80.87178	-39.515388	-53.131664	2.59209	2.370692	2.131819	1.991772	2.465886	4.437346	4.024714	3.625407	3.337044	3.85405	309	313	276	227	192	-0.008062	-0.026966	-0.075247	-0.159361	0.105094	47	57	43	45	25	3.5	3.5	3.5	3.5	3.5	19.550097	19.144545	19.025612	18.991989	18.999947	0.002679	0.002444	0.003332	0.006139	0.018784	19.554905	19.170897	19.057899	19.020786	19.050386	0.002125	0.001795	0.001976	0.002987	0.0101	15134.27832	21987.800781	24533.269531	25304.869141	25120.095703	37.331974	49.490818	75.26889	143.03923	434.480194	15067.393555	21460.566406	23814.425781	24642.541016	23979.802734	29.485754	35.48933	43.340797	67.789543	223.086365	0.983785	0.985569	0.983967	0.957384	0.913822	-0.0	0.000423	0.000273	-0.000546	-0.003511	0.000251	0.00018	0.000055	0.000071	0.000091	0.000073	0.000059	0.000069	0.000105	0.000467	0.000079	0.000067	0.000075	0.000122	0.000492	1636148068921376768	Gaia DR3 2738666826640142976	2738666826640142976	1123567962	2016.0	0.310523	0.366047	1.81633	0.190502	-0.110585	0.401154	-0.275668	8.156886	-2.322163	0.454194	-7.819358	0.199079	0.222702	0.48238	-0.134915	-0.314749	0.011064	-0.267045	0.498465	0.038252	-0.356448	0.229138	347	0	346	1	0.691306	376.412659	0.0	0.0	31	False	1.53868	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	39	16	0.637004	47	6	0	0.062898	63.298626	0	0	1.025602	0.461014	0.666481	0.316866	0.329874	-23.835056	-13.264997	-19.398878	-2.067758	False	386	387.36797	0.893934	433.32959	19.217058	43	190.928085	6.549115	29.153265	19.636368	43	241.148956	8.413721	28.661392	18.792183	1.115418	0	0	0	0	0	0.844185	0.41931	0.424875	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	NOT_AVAILABLE	98.274515	-58.631569	1.007559	1.542896	False	False	0	False	False	False	False	False	False	False	False	0.0	0.0	0.999962	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	0.007574
1369333563259788683	1033369594	DES0000+0209	4864	19459	4981606	79705703	1275291259	0.234549	1.80234	0.234549	1.80234	98.129158	-58.616264	1970.538398	127.709064	2.538163	0.026947	2.283103	0.019319	63.260887	-25.967926	3.5	0.02142	22.131264	20.782042	19.909607	19.497562	19.381517	22.125305	20.741446	19.891855	19.461432	19.368523	0	0	2	0	7	2	0	6	2	0	7	2	0	7	2	0	5	1970.561443	1970.566382	1970.546942	1970.525804	1970.524173	127.801409	127.744037	127.718423	127.69412	127.764459	1.87465	1.495105	1.248391	1.195136	1.67643	2.019633	1.527058	1.262459	1.188647	1.753073	0.054399	0.001435	-0.012565	-0.013933	0.00888	0.001359	0.000138	0.000052	0.000076	0.001434	0.001364	0.000139	0.000053	0.000076	0.001436	0.000002	-0.0	-0.0	-0.0	-0.0	1.427506	1.235768	1.126865	1.09827	1.32442	1.362538	1.222718	1.114013	1.085166	1.294378	71.557449	87.433449	-59.620884	-38.445477	83.47699	0.036943	0.011771	0.007252	0.008745	0.037889	0.03686	0.011758	0.007241	0.008734	0.037861	71.301353	-88.632751	-55.471584	-39.835339	-84.530228	2.433985	2.12629	1.974429	1.924895	2.352394	4.193262	3.654328	3.238147	3.115396	3.647523	127	149	175	166	131	0.016099	-0.04238	-0.126929	-0.10779	0.195411	50	29	36	38	42	3.5	3.5	3.5	3.5	3.5	22.199509	20.827881	19.943214	19.52318	19.403965	0.026057	0.010371	0.007321	0.009617	0.028927	22.19355	20.787285	19.925463	19.487049	19.39097	0.014439	0.005871	0.00363	0.004424	0.016302	1318.854004	4664.958496	10536.936523	15514.138672	17314.658203	31.644367	44.549587	71.031029	137.38678	461.192688	1326.111572	4842.682617	10710.623047	16039.106445	17523.134766	17.636395	26.188999	35.808666	65.359222	263.108673	0.957401	0.982467	0.986213	0.983961	0.915406	-0.001923	-0.000371	-0.000615	0.000394	-0.001991	-0.000344	0.000335	-0.000273	-0.000539	-0.000838	0.000564	0.000205	0.000141	0.000166	0.000785	0.000549	0.000229	0.000145	0.000185	0.000778	1636148068921376768	Gaia DR3 2738667101518800640	2738667101518800640	1292959389	2016.0	0.234551	1.733652	1.802338	0.904359	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	-0.377451	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	176	0	175	1	2.033548	209.553848	0.0	0.0	3	False	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	20	11	4.416066	24	4	0	0.042665	5.021078	0	0	<NA>	0.195942	0.592804	0.178633	0.210043	-33.038357	-11.288143	-30.245949	11.789377	False	199	102.490201	0.960867	106.66433	20.660662	14	47.255432	13.063424	3.617385	21.152412	23	91.423809	10.170794	8.988857	19.845247	1.353098	0	0	0	0	1	1.307165	0.491751	0.815414	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	NOT_AVAILABLE	98.129158	-58.616268	0.932301	1.560274	False	False	0	False	False	False	False	False	False	False	False	0.00002	0.000009	0.999864	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	0.010084
1369333662580669435	1033369151	DES0000+0209	4864	19459	4981606	79705709	1275291352	0.224895	1.805816	0.224895	1.805816	98.114464	-58.609436	2102.632332	175.276932	2.488673	0.012774	2.404403	0.012187	0.385307	71.260864	3.5	0.02164	19.409544	19.092943	19.013659	18.970081	19.054096	19.391396	19.066992	18.989605	18.967628	19.003067	0	0	3	0	7	3	0	6	3	0	7	3	0	7	3	0	5	2102.642949	2102.623636	2102.646911	2102.623833	2102.585853	175.196428	175.258195	175.25623	175.310274	175.337099	1.934322	1.521593	1.233268	1.118132	1.566162	2.022033	1.539527	1.253916	1.12195	1.569736	-0.007658	0.001508	-0.00673	-0.00667	0.023899	0.000018	0.000011	0.000012	0.000023	0.00062	0.000018	0.000011	0.000012	0.000023	0.000616	-0.0	0.0	-0.0	-0.0	0.0	1.422215	1.240828	1.120676	1.061593	1.261711	1.39056	1.233478	1.109625	1.055037	1.242571	-85.047653	85.225716	-73.449753	-52.985535	47.138199	0.004286	0.003291	0.003501	0.004837	0.024894	0.004247	0.003271	0.003494	0.004822	0.024816	-87.157486	85.57	-73.578835	-80.180496	2.130895	2.43521	2.160697	1.956783	1.878343	2.238704	4.184096	3.590041	3.217377	3.00365	3.430683	275	275	233	225	153	0.006937	-0.043151	-0.119704	-0.153986	0.19372	44	33	48	34	27	3.5	3.5	3.5	3.5	3.5	19.478489	19.139254	19.047611	18.995962	19.076775	0.002447	0.002425	0.003319	0.005582	0.020941	19.46034	19.113302	19.023558	18.993509	19.025745	0.002038	0.001834	0.001891	0.002974	0.012801	16166.074219	22095.226562	24041.150391	25212.462891	23404.009766	36.426945	49.34729	73.47007	129.599136	451.292389	16438.556641	22629.716797	24579.714844	25269.490234	24530.242188	30.85247	38.216866	42.811005	69.21801	289.22937	0.983661	0.987556	0.984192	0.984377	0.917405	-0.001149	-0.000591	-0.000593	-0.001037	-0.003283	0.000266	-0.000099	-0.000119	-0.000178	-0.000417	0.000068	0.000058	0.00007	0.000095	0.000544	0.000074	0.00007	0.000074	0.000125	0.000595	1636148068921376768	Gaia DR3 2738667307676475008	2738667307676475008	1701903094	2016.0	0.224895	0.362803	1.805814	0.189897	0.497227	0.326919	1.520949	7.001029	5.349427	0.53702	-4.516419	0.183331	-0.202557	0.118688	0.383355	-0.426254	-0.130372	-0.569081	0.543456	0.053851	-0.353238	-0.156056	351	0	348	3	0.344074	368.6716	0.0	0.0	31	False	1.545131	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	40	14	0.768313	50	8	0	0.045563	38.980244	0	0	1.01219	0.299254	0.657849	0.333298	0.293219	-20.634478	-11.585686	-18.034212	0.692067	False	415	416.141298	0.903141	460.770874	19.139265	46	213.388601	6.591115	32.375187	19.515614	48	258.287325	6.501014	39.730316	18.717638	1.133451	0	0	0	1	0	0.797976	0.376348	0.421627	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	NOT_AVAILABLE	98.114462	-58.609437	0.924827	1.567304	False	False	0	False	False	False	False	False	False	False	False	0.0	0.0	0.999954	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	<NA>	0.005852

5 rows × 368 columns

5.1. Plot the crossmatch points#

We will select small “boxes” in the sky to load only small parts of the catalogs and plot them.

Here we do not load the saved cross-matched catalog, but virtually cross-match the catalogs on the fly each time we run .compute(). While it would be more performant to load the cross-matched catalog, the following cell demonstrates that you can work with the cross-matching results without saving them to disk.

[16]:

ra_range = [0.0, 0.1]
dec_range = [2.45, 2.55]

des_box = des_catalog.box_search(ra=ra_range, dec=dec_range).compute()
gaia_box = gaia_catalog.box_search(ra=ra_range, dec=dec_range).compute()
xmatch_box = xmatched.box_search(ra=ra_range, dec=dec_range).compute()

ra_des = np.where(des_box["RA"] > 180, des_box["RA"] - 360, des_box["RA"])
ra_gaia = np.where(gaia_box["ra"] > 180, gaia_box["ra"] - 360, gaia_box["ra"])
ra_x_gaia = np.where(xmatch_box["ra_gaia"] > 180, xmatch_box["ra_gaia"] - 360, xmatch_box["ra_gaia"])

plt.figure(figsize=(5, 5))
plt.scatter(ra_gaia, gaia_box["dec"], s=50, alpha=1.0, color="green", label="Gaia")
plt.scatter(ra_x_gaia, xmatch_box["dec_gaia"], s=10, alpha=1.0, color="red", label="x-matched")
plt.scatter(ra_des, des_box["DEC"], s=10, alpha=0.4, marker="+", color="blue", label="DES")
plt.xlabel("RA")
plt.ylabel("Dec")
plt.legend()

[16]:

<matplotlib.legend.Legend at 0x7fdb94544df0>

../../_images/tutorials_pre_executed_des-gaia_34_1.png

5.2. Plot the joint HR diagram#

Now we may plot the color-magnitude diagram for all the DES stars using Gaia parallaxes. Instead of redoing the cross-match, we will load the saved cross-matched catalog with few columns only.

[17]:

# Read the cross-matched catalog, just like we did before for Gaia and DES,
# but keeping few columns only.
xmatched_from_disk = lsdb.open_catalog(
    XMATCH_DIR,
    columns=[
        "parallax_gaia",
        "parallax_over_error_gaia",
        "WAVG_MAG_PSF_G_des",
        "WAVG_MAG_PSF_R_des",
    ],
)

# Apply quality flags
filtered = xmatched.query(
    "parallax_over_error_gaia > 10.0 and WAVG_MAG_PSF_G_des > 0.0 and WAVG_MAG_PSF_R_des > 0.0"
)

# Get Dask data frame from the catalog object
ddf = filtered._ddf

# get 2D histogram of WAVG_MAG_PSF_R-WAVG_MAG_PSF_I vs WAVG_MAG_PSF_I, lazily
color = ddf["WAVG_MAG_PSF_G_des"] - ddf["WAVG_MAG_PSF_R_des"]
absolute_r = ddf["WAVG_MAG_PSF_R_des"] + 5 * np.log10(ddf["parallax_gaia"] / 100)

x_bins = np.linspace(-1, 2.5, 101)
y_bins = np.linspace(0, 20, 101)

hist2d = dask.array.histogram2d(color.to_dask_array(), absolute_r.to_dask_array(), bins=[x_bins, y_bins])

# Run the computation with Dask client, it will take a while
hist2d = hist2d[0].compute()

# Plot the 2D histogram
plt.imshow(
    hist2d.T,
    extent=(x_bins[0], x_bins[-1], y_bins[0], y_bins[-1]),
    aspect="auto",
    origin="lower",
    norm=LogNorm(vmin=1, vmax=hist2d.max()),
)
plt.gca().invert_yaxis()
plt.colorbar(label="Number of stars")
plt.xlabel("$g-r$")
plt.ylabel("$M_r$")
plt.title("Absolute magnitude — color diagram")

/astro/users/kmalanch/.virtualenvs/default/lib/python3.10/site-packages/dask/dataframe/dask_expr/_collection.py:1430: UserWarning: Dask currently has limited support for converting pandas extension dtypes to arrays. Converting double[pyarrow] to object dtype.
  warnings.warn(
/astro/users/kmalanch/.virtualenvs/default/lib/python3.10/site-packages/dask/dataframe/dask_expr/_collection.py:1430: UserWarning: Dask currently has limited support for converting pandas extension dtypes to arrays. Converting double[pyarrow] to object dtype.
  warnings.warn(

[17]:

Text(0.5, 1.0, 'Absolute magnitude — color diagram')

../../_images/tutorials_pre_executed_des-gaia_36_2.png

And this is how it would look like for the entire Gaia + DES cross-matched catalog. This shows ~3.6 million DES stars with Gaia parallaxes left after filtering.

HR Diagram for the entire catalogs

[18]:

client.close()

About#

Authors: Konstantin Malanchev

Last updated on: Oct 27, 2025

If you use lsdb for published research, please cite following instructions.