README
simple-url-cache
Conditionally cache your URL's content on REDIS with RegExp. Also supports cache instance sharing and isolation.
Installation
npm install simple-url-cache
API
CacheEngine
- methods
- static properties
CacheStorage
- setters/getters
- methods
Configuration
Cache Engine
setters /getters
constructor
constructor( defaultDomain: string, instanceName: string, storageConfig: Object, cacheConfig: Object)
defaultDomain Every URL that miss a hostname will get classified under this domain
instanceName The isolated instance where this cacheEngine will store urls. If another cacheEngine has the same storage type and the same instance name, they will share the pool.
storageConfig Redis Storage Config Defines how & where url content is stored
cacheConfig Cache config Supports TTL and inclusion/exclusion for any URL rules you need
Example:
var CacheEngine = require('simple-url-cache');
var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules);
var engine2 = new CachEngine('http://localhost:4444', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules);
var engine3 = new CachEngine('http://localhost:5555', 'I2', {host: '127.0.0.1',port: 6379}, cacheRules);
// At this stage, engine1 and engine2 share the same pool.
engine1.url('http://a.com/index.html').set('some content');
// resolve(true)
engine2.url('http://b.com/index.html').set('some content');
// resolve(true)
engine1.url('http://b.com/index.html').set('some content');
// resolves(false) - already cached
engine3.url('http://a.com/index.html').set('some content');
// resolve(true)
engine1.url('http://b.com/index.html').get()
//resolve(true) -> shared pool with engine1
engine3.url('http://b.com/index.html').get()
// reject(false) -> not set
url
url(url: string): CacheStorage
url Initialize a new CacheStorage instance ready to be get(), set(), delete() and has().
clearDomain
clearDomain(domain: string): Promise<boolean>
Delete all the cached urls stored within this instance under the specified domain.
clearInstance
clearInstance(): Promise<boolean>
Removes all the cached URLs for all domains for this instance.
getStoredHostnames
getAllCachedURL(): Promise<string[]>
Retrieves an array of all the domains cached.
example:
var CacheEngine = require('simple-url-cache');
var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules);
engine1.url('http://a.com/index.html').set('content').then( ... )
engine1.url('http://b.com/index.html').set('content').then( ... )
CacheEngine.getStoredHostnames().then(function(results) {
console.log(results);
// ['http://a.com', 'http://b.com']
});
domain if none provided, then the default domain will be used
getStoredURLs
getCachedDomains(idomain:string): Promise<string[]>
Get the array of cached URLs associated with this domain & instance
domain All the stored URLs retrived had this domain prepended
example:
var CacheEngine = require('simple-url-cache');
var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules);
engine1.url('http://a.com/index.html').set('content').then( ... )
engine1.url('http://a.com/about.html').set('content').then( ... )
CacheEngine.getStoredURLs().then(function(results) {
console.log(results);
// ['/index.html', '/about.html']
});
Static helper
The methods used to validate the CacheConfig and the RedisStorageConfig objects are exposed statically.
They all throw aTypeError
when invalid
validateCacheConfig()
validateCacheConfig(config: CacheRules)
validateRedisStorageConfig()
validateRedisStorageConfig(config: RedisStorageConfig)
CacheStorage
geters & setters
delete
delete(): Promise<boolean>
Resolve to true if the url has been suppressed, false if the url wasn't cached Reject an Error if any
get
get(): Promise<string>
Resolve to the url's content Reject if the url wasn't cached
has
has(): Promise<boolean>
Resolve to true if the url is cached, false if the url is not cached, rejected on error
set
set(content: string [, force: boolean]) : Promise<boolean>
Resolve to true if the url has been cached successfully,
Rejects false if
- the url matches the never
rule.
- The url has already been cached
Rejects on Error
html: the content of the url to be cached, must be UTF8
force:
- Actualize the TTL for maxAge already cached urls
- Force the caching for url matching the never
rule.
methods
getCategory()
Returns the url's internal category name. always
, maxAge
or never
getDomain()
Returns the domain which the URL has been stored with.
var url = CacheEngine.url('http://a.com/index.html');
url.set('content').then()
url.getDomain() // http://a.com
getInstanceName()
The instanceName set when this url has been stored
var CacheEngine = require('simple-url-cache');
var engine1 = new CachEngine('http://localhost:3333', 'I1', {host: '127.0.0.1',port: 6379}, cacheRules);
var engine2 = new CachEngine('http://localhost:3333', 'I2', {host: '127.0.0.1',port: 6379}, cacheRules);
var url1 = engine1.url('http://a.com/index.html')
var url2 = engine1.url('http://a.com/about.html')
url1.getInstanceName() // I1
url2.getInstanceName() // I2
getStorageType()
Same as getInstanceName()
, will return redis
Storage engines
So far, only redis is supported, but it is not hard to add more, PR are welcome.
Initially, FileSystem storage was supported, but it has been removed for several reasons :
- Performances issues.
- Huge complexity issues when dealing with large sets of data, specially when
getStoredURLs()
is called or if a power outage happens.
it had to replay the whole Regex test against each stored URL, and then make a stat on the file in case it matches a maxAge rule to check the creation time.
But if you need to add another storage engine, like mongo for example, the code is designed in a way were the CacheStorage
and CacheEngine
APIs are completly storage independent.
Config Files
Cache Config
This is an object describing which URL will be cached, which URLs won't be cached, and which ones will have a ttl expiration.
This is the same object, independently of the storage engine used.
An example worth 1000 words :
exports.cacheConfig = {
// Will cache all URL starting with /posts/ and ending with html for 24 hours
cacheMaxAge: [
{
regex: /^\/posts.*html$/,
maxAge: 3600
}
],
// Will cache about-us.html, contact-us.html and /prices.html indefinitively
cacheAlways: [
{
regex: /^about-us\.html$/,
regex: /^contact-us\.html$/,
regex: /^prices\.html$/
}
],
// will never cache the url /sitemaps.html
cacheNever: [
{
regex: /^sitemaps\.html$/
}
],
// If no URL is matched against these rules, then the default is to never cache it. can be 'never' or 'always'
default: 'never'
};
Redis storage config
A bit more complex. The library noderedis is used here, so a valid redis node config file is needed.
example :
export.redisStorageConfig = {
host: '127.0.0.1',
port: 6379,
socket_keepalive: true
}