valuepack-mine-npm

Mines the npm registry for user and package data used by valuepack.

Usage no npm install needed!

<script type="module">
  import valuepackMineNpm from 'https://cdn.skypack.dev/valuepack-mine-npm';
</script>

README

valuepack-mine-npm

Supports valuepack, the community driven rating system for nodejs modules on npm in order to help in selecting the right one.

Read more about its goals.

scripts

You can play with scripts inside ./scripts.

environment variables

The following environment variables are considered by the scripts:

  • VALUEPACK_DATA the directory in which json data fetched from the npm registry is stored (defaults to valuepack-mine-npm/data)
  • VALUEPACK_MINE_DB the path at which the leveldb data is stored (defaults to valuepack-mine-npm/store/valuepack-mine.db)

Make sure to include them every time you execute a script or add the following to your .bashrc (example):

export VALUEPACK_DATA=~/.valuepack/data
export VALUEPACK_MINE_DB=~/.valuepack/valuepack-mine.db

fetching data initializing the data store

An init script is provided that fetches all needed data from npm and stores it as json and in a leveldb database at ~/.valuepack.

Do one of the following:

npm run init or cd scripts && ./init.sh

fetch npm data scripts

# fetch users
./fetch-npm-users.js

# fetch packages
./fetch-npm-packages.js

rebuild script

In order to store all data in leveldb, please run:

./rebuild-store.js

Note: rebuilding the database from local json files takes about 2mins on a MacBookAir.

store scripts

All store-* scripts take parameters.

  • --read read out values of the store instead of adding them
  • --keys pull out and print keys only
  • --values pull out and print values only

Therfore if you want to query data you should always pass the --read flag.

store-npm-packages specific parameters

  • --owners list owner indexes instead of packages
  • --keyword list keyword indexes instead of packages

store-npm-users specific parameters

  • --github list github login indexes instead of users

examples

# get package keyword count
./store-npm-packages.js --read --keyword --keys | wc -l

# get package count
./store-npm-packages.js --read --keys | wc -l

# query how many users did provide a github account
./store-npm-users.js --read --github --keys | wc -l

# query how many packages dominictarr published
./store-npm-packages.js --read --owner --keys | grep dominictarr | wc -l