node-pyspark

This module brings Apache Spark API to nodejs.

WARNING This package is still in its early stages of development, and not all pyspark APIs have been ported yet.

Usage

The API is very similar to the pyspark API with some notable differences:

• All Functions and methods take an object argument. The keys of which represent the function parameters in pyspark
• Most functions are synchronous (in perception), but some return a promise and can be awaited upon

// importing is similar to that in pyspark
const { SparkSession, DataFrame, types, } = require('./index').sql;

// create spark session
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()

// create a dataframe
df = spark.createDataFrame({ data: [1, 2, 3], schema: types.IntegerType() })

// show returns a promise
await df.show()

// stop the sessions
spark.stop()

APIs implemented

pyspark APIs that have been ported to node-pyspark

Class/Object	API	Comments
Builder	appName
Builder	config
Builder	appName
Builder	enableHiveSupport
Builder	master
Builder	getOrCreate
SparkSession	range
SparkSession	createDataFrame
SparkSession	sql
SparkSession	stop
SparkSession	table
SparkSession	put	non standard API
SparkSession	wait	non standard API
SparkSession	_exec
SparkSession	_eval
DataFrameReader	csv
DataFrameReader	format
DataFrameReader	jdbc
DataFrameReader	json
DataFrameReader	load
DataFrameReader	option
DataFrameReader	orc
DataFrameReader	parquet
DataFrameReader	schema
DataFrameReader	table
DataFrameReader	text
DataFrameWriter	bucketBy
DataFrameWriter	csv
DataFrameWriter	format
DataFrameWriter	insertInto
DataFrameWriter	jdbc
DataFrameWriter	json
DataFrameWriter	mode
DataFrameWriter	option
DataFrameWriter	orc
DataFrameWriter	parquet
DataFrameWriter	partitionBy
DataFrameWriter	save
DataFrameWriter	saveAsTable
DataFrameWriter	sortBy
DataFrameWriter	text
DataFrame	agg
DataFrame	alias
DataFrame	approxQuantile
DataFrame	cache
DataFrame	checkpoint
DataFrame	coalesce
DataFrame	_exec
DataFrame	_eval