[Climbing] 10/01/22 - 10/17/22: Red River Gorge

-





























 

Comments

Post a Comment

Popular posts from this blog

Benchmarks: Dask Distributed vs. Ray for Dask Workloads

-
Image
  Dask Workloads  In this section, I detail the dask workloads that will be used to benchmark two cluster backends - Ray and Dask Distributed. In summary, my dask workload involves processing ~3.3 Petabytes of ~6.06 million input files with dask-based Processing Chain and producing ~14.88 Terabytes of ~1.86 million output files. Input Data ~6.06 million ~550MB files, totaling ~3.3 Petabytes  Each file represents a N-dimensional numerical tensor (np.ndarray).  Processing Chain: How we go from Input data to Output data (Dask Task Graphs) We use  Xarray  / Dask APIs for  lazy , numerical computations on out-of-core multidimensional tensors. Per 3 input files, we generate 1 output file using below  Dask Task Graph  (auto generated by using Dask APIs) The Task Graph is consisted of small, Dask-level 'tasks'; tasks can be grouped into 3 actions:   Load Download file from S3 and load each file into 2D  np.ndarray  of type np.floa...
Read more

[10/29/22 - 11/13/22] Siurana

-
Image
Some photos on our sport climbing trip to Siurana!  On our way there ..    Our airbnb!  Climbing days -  climbing in Arboli  Rest day activity #1: Locals were gathering outside to enjoy some food on the Sunday early afternoon; we joined in  Rest day activity #2: Checking out barcelona  Food - Lots of ham  Rest day #2: Reus Rest day - view from top at Siurana town  Sandstone  Siurana  Coming back home..
Read more

Parallel processing libraries in Python - Dask vs Ray

-
At work, I'm working with enabling numerical computations/transformations of N-dimensional tensors that cannot fit in a single machine's memory. They are "big data" in the order of TBs-PBs. The data types are numerical arrays; e.g. np.array. Data processing / analysis code are written in Python. I looked at two libraries to enable parallel computing within the Python ecosystem. The two are Dask and Ray. I made some comparison between them.   Dask Dask is a flexible library for parallel computing in Python  https://docs.dask.org/en/latest/ Ray Ray is a fast and simple framework for building and running distributed applications. https://ray.readthedocs.io/en/latest/ (Originally published 02/25/2020, updated 03/03/2021 as per new features in the libraries) Dimension Dask Ray Color Code Justification Efficient Data Sharing Between Nodes  Source:  https://distributed.dask.org/en/latest/serialization.html Data sharing happens via TCP messages between workers. Every ti...
Read more