Posts

Parallel processing libraries in Python - Dask vs Ray

At work, I'm working with enabling numerical computations/transformations of N-dimensional tensors that cannot fit in a single machine's memory. They are "big data" in the order of TBs-PBs. The data types are numerical arrays; e.g. np.array. Data processing / analysis code are written in Python. I looked at two libraries to enable parallel computing within the Python ecosystem. The two are Dask and Ray. I made some comparison between them.   Dask Dask is a flexible library for parallel computing in Python  https://docs.dask.org/en/latest/ Ray Ray is a fast and simple framework for building and running distributed applications. https://ray.readthedocs.io/en/latest/ (Originally published 02/25/2020, updated 03/03/2021 as per new features in the libraries) Dimension Dask Ray Color Code Justification Efficient Data Sharing Between Nodes  Source:  https://distributed.dask.org/en/latest/serialization.html Data sharing happens via TCP messages between workers. Every ti...

[Snow Sports] (02/02/20 - 02/07/20) Snowboarding in Zermatt, Switzerland

Image
We spent 5 days skiing in Zermatt! (with a rest day in between!) Skiing details We weren't blessed with the weather the first couple days; it was stormy and very windy, so the Matterhorn side was closed. We could ski in the Sunnegga side though! We were definitely anxious to see the Matterhorn but it was nowhere to be found in those two couple cloudy days. From the fourth day (third day skiing), the weather became great, and we were able to see the Matterhorn! One day was spent just enjoying skiing near the Matterhorn glacier. Another day was spent going to and back from the Italy side. And another day was spent taking an off-piste lesson with Bruno who was great! He showed us some good off-piste run near Rothorn that we could not have found on our own. He also equipped us some avalanche transceivers although avalanche risks were pretty low that day.  My favorite part was just the mere fact that I was skiing with Matterhorn so close by. I also very much enjoyed learning abou...

[Climbing] (09/30/19) Prime Rib Route (5.9-, 11 pitches, Grade III) @ Goat Wall in Mazama, WA

Image
Route Details:  Mountain Project:  https://www.mountainproject.com/route/107730934/prime-rib Difficulty: 5.9-, 11 pitches, Grade III Report:  This summer, we did several other multi-pitch routes - a few at 2-3 pitches and one at 7 pitches ( Condorphamine Addiction @ Leavenworth). Prime Rib was definitely the longest out of all at 11 pitches. How the day went- I found that the climbing itself wasn't difficult at any sections of any pitches. All of it was very enjoyable. It was pretty cold though (low to high 40 degrees throughout the day) which added to the difficulty. My hands and feet were a bit numb at some parts of the climbs. It also had been raining so there were some wet sections which made some slab sections pretty slippery.  I was worried about walking along the ledges to find the base of pitch 3 and 6. It turned out to be no problem though; it was pretty obvious where to go, and the "ledge" actually felt more like a hiking trail with plenty room from...

[Climbing] (6/2 - 6/7/19) Taking "Alpinism 2" course and exploring the Washington Pass

Image
Course Link:  here Trip Details! Jay, Nate, Alex and I took a course in Intermediate Alpine Climbing with American Alpine Institute. I said "yo this looks fun!" about half a year ago and they were all down 😅.  We worked with 2 guides - Kevin and Samantha.  First four days, we learned Trad and Multi-pitch climbing skills and summited two peaks in the North Cascades while trad climbing and short pitching.  During these days, we camped for 4 nights in the frontcountry. Liberty Bell:  https://www.mountainproject.com/area/105797864/liberty-bell South Early Winter Spire:  https://www.mountainproject.com/area/106257377/south-early-winters-spire The last couple days - we were aiming to summit the Silver Start Mountain via Silver Start Glacier. We went to the bench camp, camped there in the backcountry on the fifth night, then got up at 3AM for the summit attempt. It was up to us to scramble up and route find in the dark. Unfortunately we got lost a bit and ended u...

[Mountaineering] (08/13/18) Mt. Rainier Summit

Image
I did Mt. Rainier summit with RMI! Program details here Route we took: https://www.mountainproject.com/route/105877100/ingraham-glacier-disappointment-cleaver-route  Motivation:  When I got to Seattle in the summer of 2017, I thought "hiking" Mt. Rainier would be an iconic thing to do. I only knew a little about mountaineering back then so I had no idea what was actually involved. I saw the RMI program online and signed up to be on the waitlist. In February 2018, while sitting at work, I got an email that I got off the waitlist. I think I was feeling particularly adventurous that day because I just thought "why the hell not!" and signed up.  Well.. when I came home and began to actually look into it (facepalm), I only then realized it was kind of a big deal! Not only were there so much mountaineering knowledge and equipment involved, which were all new to me, I realized that the fitness requirement was no joke. Well... time to train!  Training:  From...