Dask is distributed version of nunpy, pandas and scikit-learn.

File System

Dask can use local file system and HDFS. For example:

import dask.dataframe as dd
df = dd.read_csv('data/2000-*-*.csv')

import dask.bag as db
b = db.read_text('hdfs://path/to/*.json').map(json.loads)

Ref:

• Remote Data
• DataFrames: Read and Write Data
• Create and Store Dask DataFrames

Cluster Setup

By SSH.