Edit on GitHub

Setup (setup.jl)

This pipeline is written in Julia, so you will need a Julia installation in order to run the components. We recommend you use the precompiled binaries provided on the Julia website. Make sure to add the Julia executable to your path, as described in the platform-specific instructions.

Package Installation

We use the package manager in the Julia interpeter to install the latest versions of Healpix and PowerSpectra. This will be simpler in the future, when we tag a stable version of these packages for the General Registry. For now, we add the latest versions of these packages from GitHub. Note that package installation requires an internet connection, so unlike the other parts of the pipeline, setup.jl requires an internet connection. If you're on a cluster, that means you need to run this file on the head node in order to install packages.

using Pkg
Pkg.add.(["Healpix", "PowerSpectra", "CSV", "DataFrames", "TOML",
  "BlackBoxOptim", "FileIO", "JLD2", "FITSIO",
  "DataInterpolations", "Optim", "GaussianProcesses"]);
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`

The command-line interface for this basic pipeline setup script is

$ julia setup.jl global.toml
  • It displays the contents of the global TOML configuration file named global.toml.
  • This script downloads the Planck data to the specified directories in global.toml.

Configuration

All of the pipeline scripts take a configuration TOML file as the first argument. We now print out just the [dir] entry in the TOML, which is what you will need to configure.

using TOML
configfile = ARGS[1]  # read in the first command line argument
println("config filename: ", configfile, "\n")

# take a look at the config
config = TOML.parsefile(configfile)
TOML.print(Dict("dir"=>config))  # print just the "dir" TOML entry
config filename: example.toml

[dir]
scratch = "/home/zack/planck256/"

    [dir.poleff]
    P100hm1 = 0.9995
    P143hm1 = 0.999
    P217hm1 = 0.999
    P143hm2 = 0.999
    P100hm2 = 0.9995
    P217hm2 = 0.999

    [dir.map]
    P100hm1 = "nside256_HFI_SkyMap_100_2048_R3.01_halfmission-1.fits"
    P143hm1 = "nside256_HFI_SkyMap_143_2048_R3.01_halfmission-1.fits"
    P143hm2 = "nside256_HFI_SkyMap_143_2048_R3.01_halfmission-2.fits"
    P217hm1 = "nside256_HFI_SkyMap_217_2048_R3.01_halfmission-1.fits"
    P100hm2 = "nside256_HFI_SkyMap_100_2048_R3.01_halfmission-2.fits"
    P217hm2 = "nside256_HFI_SkyMap_217_2048_R3.01_halfmission-2.fits"

    [dir.maskT]
    P100hm1 = "nside256_COM_Mask_Likelihood-temperature-100-hm1_2048_R3.00.fits"
    P143hm1 = "nside256_COM_Mask_Likelihood-temperature-143-hm1_2048_R3.00.fits"
    P143hm2 = "nside256_COM_Mask_Likelihood-temperature-143-hm2_2048_R3.00.fits"
    P217hm1 = "nside256_COM_Mask_Likelihood-temperature-217-hm1_2048_R3.00.fits"
    P100hm2 = "nside256_COM_Mask_Likelihood-temperature-100-hm2_2048_R3.00.fits"
    P217hm2 = "nside256_COM_Mask_Likelihood-temperature-217-hm2_2048_R3.00.fits"

    [dir.general]
    name = "example"
    nside = 256

    [dir.maskP]
    P100hm1 = "nside256_COM_Mask_Likelihood-polarization-100-hm1_2048_R3.00.fits"
    P143hm1 = "nside256_COM_Mask_Likelihood-polarization-143-hm1_2048_R3.00.fits"
    P143hm2 = "nside256_COM_Mask_Likelihood-polarization-143-hm2_2048_R3.00.fits"
    P217hm1 = "nside256_COM_Mask_Likelihood-polarization-217-hm1_2048_R3.00.fits"
    P100hm2 = "nside256_COM_Mask_Likelihood-polarization-100-hm2_2048_R3.00.fits"
    P217hm2 = "nside256_COM_Mask_Likelihood-polarization-217-hm2_2048_R3.00.fits"

    [dir.url]
    beams = "https://irsa.ipac.caltech.edu/data/Planck/release_3/ancillary-data/"
    masks = "https://github.com/xzackli/PSPipePlanckRender.jl/releases/download/v0.1/"
    maps = "https://github.com/xzackli/PSPipePlanckRender.jl/releases/download/0.1.1/"
  • The scratch space is where intermediary files are deposited.
  • Note that each map file has an identifier. This shortens the long names, but more importantly allows one to set up a custom system of names when we cross-correlate Planck with other experiments.
  • In this case, we preface all Planck maps and masks with P, and include the frequency and split.

Downloading the Planck 2018 Data

# set up download urls
using Downloads

if length(ARGS) == 0
    TARGET_DIR = pwd()
else
    TARGET_DIR = ARGS[1]
end

# set up directories
mapdest = joinpath(config["scratch"], "maps")
maskdest = joinpath(config["scratch"], "masks")
beamdest = joinpath(config["scratch"], "beams")
mkpath(mapdest)
mkpath(maskdest)
mkpath(beamdest)

function download_if_necessary(url, dest; verbose=true)
    if isfile(dest) == false
        verbose && println("Downloading ", dest)
        @time Downloads.download(url, dest)
    else
        verbose && println("Extant, skip ", dest)
    end
end
download_if_necessary (generic function with 1 method)
# now read from config and then actually download
mapfiles = values(config["map"])
maskfiles = [values(config["maskT"])..., values(config["maskP"])...]

for f in mapfiles
    download_if_necessary(joinpath(config["url"]["maps"], f), joinpath(mapdest, f))
end

for f in maskfiles
    download_if_necessary(joinpath(config["url"]["masks"], f), joinpath(maskdest, f))
end

# just download beamfile to the target directory base.
beamfile = "HFI_RIMO_BEAMS_R3.01.tar.gz"
fullbeamfile = joinpath(beamdest, beamfile)
download_if_necessary(joinpath(config["url"]["beams"], beamfile), fullbeamfile)
run(`cp $(fullbeamfile) $(joinpath(beamdest, "tempbeamgz"))`)
run(`gzip -f -d $(joinpath(beamdest, beamfile))`)
run(`mv $(joinpath(beamdest, "tempbeamgz")) $(fullbeamfile)`)
beamfiletar = replace(beamfile, ".tar.gz"=>".tar")
run(`tar -xf $(joinpath(beamdest, beamfiletar)) --overwrite -C $(beamdest)`);
Extant, skip /home/zack/planck256/maps/nside256_HFI_SkyMap_100_2048_R3.01_halfmission-1.fits
Extant, skip /home/zack/planck256/maps/nside256_HFI_SkyMap_143_2048_R3.01_halfmission-1.fits
Extant, skip /home/zack/planck256/maps/nside256_HFI_SkyMap_143_2048_R3.01_halfmission-2.fits
Extant, skip /home/zack/planck256/maps/nside256_HFI_SkyMap_217_2048_R3.01_halfmission-1.fits
Extant, skip /home/zack/planck256/maps/nside256_HFI_SkyMap_100_2048_R3.01_halfmission-2.fits
Extant, skip /home/zack/planck256/maps/nside256_HFI_SkyMap_217_2048_R3.01_halfmission-2.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-temperature-100-hm1_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-temperature-143-hm1_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-temperature-143-hm2_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-temperature-217-hm1_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-temperature-100-hm2_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-temperature-217-hm2_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-polarization-100-hm1_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-polarization-143-hm1_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-polarization-143-hm2_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-polarization-217-hm1_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-polarization-100-hm2_2048_R3.00.fits
Extant, skip /home/zack/planck256/masks/nside256_COM_Mask_Likelihood-polarization-217-hm2_2048_R3.00.fits
Extant, skip /home/zack/planck256/beams/HFI_RIMO_BEAMS_R3.01.tar.gz
# We also want to retrieve the plic templates.
plicrefdir = joinpath(config["scratch"])
plicreffile = joinpath(config["scratch"], "plicref.tar.gz")
download_if_necessary(
    "https://github.com/xzackli/PSPipePlanckRender.jl/releases/download/0.1.2/plicref.tar.gz",
    plicreffile)
run(`tar -xzvf $(plicreffile) --overwrite -C $(plicrefdir)`);
Extant, skip /home/zack/planck256/plicref.tar.gz
plicref/
plicref/base_plikHM_TTTEEE_lowl_lowE_lensing.minimum.plik_foregrounds
plicref/covmat.dat
plicref/data_extracted.dat
plicref/EE_e2e_cnoise.ermform.dat
plicref/foreground_cl.txt
plicref/leakage.ermform.dat
plicref/subpix.ermform.dat
plicref/theory_cl.txt
plicref/vec_all_spectra.dat

For the sims, we will use the full pixel weights to efficiently compute the spherical harmonic transforms. The pixel weights are lazily downloaded in the Healpix package. We trigger that downloading here. Similarly we trigger some downloading of example data from PowerSpectra.jl for the purpose of plotting.

using Healpix
Healpix.applyFullWeights!(HealpixMap{Float64, RingOrder}(2048))
using PowerSpectra
PowerSpectra.planck256_beamdir()
"/home/zack/.julia/artifacts/4e546e76ad331ddacfd39a18d8e3e33d065a0179"