main funcions fixes

This commit is contained in:
2025-09-29 22:06:11 +09:00
parent 40e016e128
commit c8c3274527
7995 changed files with 1517998 additions and 1057 deletions

View File

@@ -0,0 +1,16 @@
ISC License
Copyright 2017-2022 (c) npm, Inc.
Permission to use, copy, modify, and/or distribute this software for
any purpose with or without fee is hereby granted, provided that the
above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE COPYRIGHT HOLDER DISCLAIMS
ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE
COPYRIGHT HOLDER BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR
CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE
USE OR PERFORMANCE OF THIS SOFTWARE.

View File

@@ -0,0 +1,361 @@
# make-fetch-happen
[![npm version](https://img.shields.io/npm/v/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![license](https://img.shields.io/npm/l/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![Travis](https://img.shields.io/travis/npm/make-fetch-happen.svg)](https://travis-ci.org/npm/make-fetch-happen) [![Coverage Status](https://coveralls.io/repos/github/npm/make-fetch-happen/badge.svg?branch=latest)](https://coveralls.io/github/npm/make-fetch-happen?branch=latest)
[`make-fetch-happen`](https://github.com/npm/make-fetch-happen) is a Node.js
library that wraps [`minipass-fetch`](https://github.com/npm/minipass-fetch) with additional
features [`minipass-fetch`](https://github.com/npm/minipass-fetch) doesn't intend to include, including HTTP Cache support, request
pooling, proxies, retries, [and more](#features)!
## Install
`$ npm install --save make-fetch-happen`
## Table of Contents
* [Example](#example)
* [Features](#features)
* [Contributing](#contributing)
* [API](#api)
* [`fetch`](#fetch)
* [`fetch.defaults`](#fetch-defaults)
* [`minipass-fetch` options](#minipass-fetch-options)
* [`make-fetch-happen` options](#extra-options)
* [`opts.cachePath`](#opts-cache-path)
* [`opts.cache`](#opts-cache)
* [`opts.proxy`](#opts-proxy)
* [`opts.noProxy`](#opts-no-proxy)
* [`opts.ca, opts.cert, opts.key`](#https-opts)
* [`opts.maxSockets`](#opts-max-sockets)
* [`opts.retry`](#opts-retry)
* [`opts.onRetry`](#opts-onretry)
* [`opts.integrity`](#opts-integrity)
* [`opts.dns`](#opts-dns)
* [Message From Our Sponsors](#wow)
### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-cache' // path where cache will be written (and read)
})
fetch('https://registry.npmjs.org/make-fetch-happen').then(res => {
return res.json() // download the body as JSON
}).then(body => {
console.log(`got ${body.name} from web`)
return fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'no-cache' // forces a conditional request
})
}).then(res => {
console.log(res.status) // 304! cache validated!
return res.json().then(body => {
console.log(`got ${body.name} from cache`)
})
})
```
### Features
* Builds around [`minipass-fetch`](https://npm.im/minipass-fetch) for the core [`fetch` API](https://fetch.spec.whatwg.org) implementation
* Request pooling out of the box
* Quite fast, really
* Automatic HTTP-semantics-aware request retries
* Cache-fallback automatic "offline mode"
* Proxy support (http, https, socks, socks4, socks5)
* Built-in request caching following full HTTP caching rules (`Cache-Control`, `ETag`, `304`s, cache fallback on error, etc).
* Node.js Stream support
* Transparent gzip and deflate support
* [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) support
* Literally punches nazis
* Built in DNS cache
* (PENDING) Range request caching and resuming
### Contributing
The make-fetch-happen team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The [Contributor Guide](https://github.com/npm/cli/blob/latest/CONTRIBUTING.md) outlines the process for community interaction and contribution. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.
All participants and maintainers in this project are expected to follow the [npm Code of Conduct](https://www.npmjs.com/policies/conduct), and just generally be excellent to each other.
Please refer to the [Changelog](CHANGELOG.md) for project history details, too.
Happy hacking!
### API
#### <a name="fetch"></a> `> fetch(uriOrRequest, [opts]) -> Promise<Response>`
This function implements most of the [`fetch` API](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch): given a `uri` string or a `Request` instance, it will fire off an http request and return a Promise containing the relevant response.
If `opts` is provided, the [`minipass-fetch`-specific options](#minipass-fetch-options) will be passed to that library. There are also [additional options](#extra-options) specific to make-fetch-happen that add various features, such as HTTP caching, integrity verification, proxy support, and more.
##### Example
```javascript
fetch('https://google.com').then(res => res.buffer())
```
#### <a name="fetch-defaults"></a> `> fetch.defaults([defaultUrl], [defaultOpts])`
Returns a new `fetch` function that will call `make-fetch-happen` using `defaultUrl` and `defaultOpts` as default values to any calls.
A defaulted `fetch` will also have a `.defaults()` method, so they can be chained.
##### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-local-cache'
})
fetch('https://registry.npmjs.org/make-fetch-happen') // will always use the cache
```
#### <a name="minipass-fetch-options"></a> `> minipass-fetch options`
The following options for `minipass-fetch` are used as-is:
* method
* body
* redirect
* follow
* timeout
* compress
* size
These other options are modified or augmented by make-fetch-happen:
* headers - Default `User-Agent` set to make-fetch happen. `Connection` is set to `keep-alive` or `close` automatically depending on `opts.agent`.
* agent
* If agent is null, an http or https Agent will be automatically used. By default, these will be `http.globalAgent` and `https.globalAgent`.
* If [`opts.proxy`](#opts-proxy) is provided and `opts.agent` is null, the agent will be set to an appropriate proxy-handling agent.
* If `opts.agent` is an object, it will be used as the request-pooling agent argument for this request.
* If `opts.agent` is `false`, it will be passed as-is to the underlying request library. This causes a new Agent to be spawned for every request.
For more details, see [the documentation for `minipass-fetch` itself](https://github.com/npm/minipass-fetch#options).
#### <a name="extra-options"></a> `> make-fetch-happen options`
make-fetch-happen augments the `minipass-fetch` API with additional features available through extra options. The following extra options are available:
* [`opts.cachePath`](#opts-cache-path) - Cache target to read/write
* [`opts.cache`](#opts-cache) - `fetch` cache mode. Controls cache *behavior*.
* [`opts.proxy`](#opts-proxy) - Proxy agent
* [`opts.noProxy`](#opts-no-proxy) - Domain segments to disable proxying for.
* [`opts.ca, opts.cert, opts.key, opts.strictSSL`](#https-opts)
* [`opts.localAddress`](#opts-local-address)
* [`opts.maxSockets`](#opts-max-sockets)
* [`opts.retry`](#opts-retry) - Request retry settings
* [`opts.onRetry`](#opts-onretry) - a function called whenever a retry is attempted
* [`opts.integrity`](#opts-integrity) - [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata.
* [`opts.dns`](#opts-dns) - DNS cache options
#### <a name="opts-cache-path"></a> `> opts.cachePath`
A string `Path` to be used as the cache root for [`cacache`](https://npm.im/cacache).
**NOTE**: Requests will not be cached unless their response bodies are consumed. You will need to use one of the `res.json()`, `res.buffer()`, etc methods on the response, or drain the `res.body` stream, in order for it to be written.
The default cache manager also adds the following headers to cached responses:
* `X-Local-Cache`: Path to the cache the content was found in
* `X-Local-Cache-Key`: Unique cache entry key for this response
* `X-Local-Cache-Mode`: Always `stream` to indicate how the response was read from cacache
* `X-Local-Cache-Hash`: Specific integrity hash for the cached entry
* `X-Local-Cache-Status`: One of `miss`, `hit`, `stale`, `revalidated`, `updated`, or `skip` to signal how the response was created
* `X-Local-Cache-Time`: UTCString of the cache insertion time for the entry
Using [`cacache`](https://npm.im/cacache), a call like this may be used to
manually fetch the cached entry:
```javascript
const h = response.headers
cacache.get(h.get('x-local-cache'), h.get('x-local-cache-key'))
// grab content only, directly:
cacache.get.byDigest(h.get('x-local-cache'), h.get('x-local-cache-hash'))
```
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen', {
cachePath: './my-local-cache'
}) // -> 200-level response will be written to disk
```
#### <a name="opts-cache"></a> `> opts.cache`
This option follows the standard `fetch` API cache option. This option will do nothing if [`opts.cachePath`](#opts-cache-path) is null. The following values are accepted (as strings):
* `default` - Fetch will inspect the HTTP cache on the way to the network. If there is a fresh response it will be used. If there is a stale response a conditional request will be created, and a normal request otherwise. It then updates the HTTP cache with the response. If the revalidation request fails (for example, on a 500 or if you're offline), the stale response will be returned.
* `no-store` - Fetch behaves as if there is no HTTP cache at all.
* `reload` - Fetch behaves as if there is no HTTP cache on the way to the network. Ergo, it creates a normal request and updates the HTTP cache with the response.
* `no-cache` - Fetch creates a conditional request if there is a response in the HTTP cache and a normal request otherwise. It then updates the HTTP cache with the response.
* `force-cache` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it creates a normal request and updates the HTTP cache with the response.
* `only-if-cached` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it returns a network error. (Can only be used when requests mode is "same-origin". Any cached redirects will be followed assuming requests redirect mode is "follow" and the redirects do not violate requests mode.)
(Note: option descriptions are taken from https://fetch.spec.whatwg.org/#http-network-or-cache-fetch)
##### Example
```javascript
const fetch = require('make-fetch-happen').defaults({
cachePath: './my-cache'
})
// Will error with ENOTCACHED if we haven't already cached this url
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'only-if-cached'
})
// Will refresh any local content and cache the new response
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'reload'
})
// Will use any local data, even if stale. Otherwise, will hit network.
fetch('https://registry.npmjs.org/make-fetch-happen', {
cache: 'force-cache'
})
```
#### <a name="opts-proxy"></a> `> opts.proxy`
A string or `new url.URL()`-d URI to proxy through. Different Proxy handlers will be
used depending on the proxy's protocol.
Additionally, `process.env.HTTP_PROXY`, `process.env.HTTPS_PROXY`, and
`process.env.PROXY` are used if present and no `opts.proxy` value is provided.
(Pending) `process.env.NO_PROXY` may also be configured to skip proxying requests for all, or specific domains.
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen', {
proxy: 'https://corporate.yourcompany.proxy:4445'
})
fetch('https://registry.npmjs.org/make-fetch-happen', {
proxy: {
protocol: 'https:',
hostname: 'corporate.yourcompany.proxy',
port: 4445
}
})
```
#### <a name="opts-no-proxy"></a> `> opts.noProxy`
If present, should be a comma-separated string or an array of domain extensions
that a proxy should _not_ be used for.
This option may also be provided through `process.env.NO_PROXY`.
#### <a name="https-opts"></a> `> opts.ca, opts.cert, opts.key, opts.strictSSL`
These values are passed in directly to the HTTPS agent and will be used for both
proxied and unproxied outgoing HTTPS requests. They mostly correspond to the
same options the `https` module accepts, which will be themselves passed to
`tls.connect()`. `opts.strictSSL` corresponds to `rejectUnauthorized`.
#### <a name="opts-local-address"></a> `> opts.localAddress`
Passed directly to `http` and `https` request calls. Determines the local
address to bind to.
#### <a name="opts-max-sockets"></a> `> opts.maxSockets`
Default: 15
Maximum number of active concurrent sockets to use for the underlying
Http/Https/Proxy agents. This setting applies once per spawned agent.
15 is probably a _pretty good value_ for most use-cases, and balances speed
with, uh, not knocking out people's routers. 🤓
#### <a name="opts-retry"></a> `> opts.retry`
An object that can be used to tune request retry settings. Retries will only be attempted on the following conditions:
* Request method is NOT `POST` AND
* Request status is one of: `408`, `420`, `429`, or any status in the 500-range. OR
* Request errored with `ECONNRESET`, `ECONNREFUSED`, `EADDRINUSE`, `ETIMEDOUT`, or the `fetch` error `request-timeout`.
The following are worth noting as explicitly not retried:
* `getaddrinfo ENOTFOUND` and will be assumed to be either an unreachable domain or the user will be assumed offline. If a response is cached, it will be returned immediately.
If `opts.retry` is `false`, it is equivalent to `{retries: 0}`
If `opts.retry` is a number, it is equivalent to `{retries: num}`
The following retry options are available if you want more control over it:
* retries
* factor
* minTimeout
* maxTimeout
* randomize
For details on what each of these do, refer to the [`retry`](https://npm.im/retry) documentation.
##### Example
```javascript
fetch('https://flaky.site.com', {
retry: {
retries: 10,
randomize: true
}
})
fetch('http://reliable.site.com', {
retry: false
})
fetch('http://one-more.site.com', {
retry: 3
})
```
#### <a name="opts-onretry"></a> `> opts.onRetry`
A function called whenever a retry is attempted.
##### Example
```javascript
fetch('https://flaky.site.com', {
onRetry() {
console.log('we will retry!')
}
})
```
#### <a name="opts-integrity"></a> `> opts.integrity`
Matches the response body against the given [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata. If verification fails, the request will fail with an `EINTEGRITY` error.
`integrity` may either be a string or an [`ssri`](https://npm.im/ssri) `Integrity`-like.
##### Example
```javascript
fetch('https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // -> ok
fetch('https://malicious-registry.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
}) // Error: EINTEGRITY
```
#### <a name="opts-dns"></a> `> opts.dns`
An object that provides options for the built-in DNS cache. The following options are available:
Note: Due to limitations in the current proxy agent implementation, users of proxies will not benefit from the DNS cache.
* `ttl`: Milliseconds to keep cached DNS responses for. Defaults to `5 * 60 * 1000` (5 minutes)
* `lookup`: A custom lookup function, see [`dns.lookup()`](https://nodejs.org/api/dns.html#dnslookuphostname-options-callback) for implementation details. Defaults to `require('dns').lookup`.

View File

@@ -0,0 +1,214 @@
'use strict'
const LRU = require('lru-cache')
const url = require('url')
const isLambda = require('is-lambda')
const dns = require('./dns.js')
const AGENT_CACHE = new LRU({ max: 50 })
const HttpAgent = require('agentkeepalive')
const HttpsAgent = HttpAgent.HttpsAgent
module.exports = getAgent
const getAgentTimeout = timeout =>
typeof timeout !== 'number' || !timeout ? 0 : timeout + 1
const getMaxSockets = maxSockets => maxSockets || 15
function getAgent (uri, opts) {
const parsedUri = new url.URL(typeof uri === 'string' ? uri : uri.url)
const isHttps = parsedUri.protocol === 'https:'
const pxuri = getProxyUri(parsedUri.href, opts)
// If opts.timeout is zero, set the agentTimeout to zero as well. A timeout
// of zero disables the timeout behavior (OS limits still apply). Else, if
// opts.timeout is a non-zero value, set it to timeout + 1, to ensure that
// the node-fetch-npm timeout will always fire first, giving us more
// consistent errors.
const agentTimeout = getAgentTimeout(opts.timeout)
const agentMaxSockets = getMaxSockets(opts.maxSockets)
const key = [
`https:${isHttps}`,
pxuri
? `proxy:${pxuri.protocol}//${pxuri.host}:${pxuri.port}`
: '>no-proxy<',
`local-address:${opts.localAddress || '>no-local-address<'}`,
`strict-ssl:${isHttps ? opts.rejectUnauthorized : '>no-strict-ssl<'}`,
`ca:${(isHttps && opts.ca) || '>no-ca<'}`,
`cert:${(isHttps && opts.cert) || '>no-cert<'}`,
`key:${(isHttps && opts.key) || '>no-key<'}`,
`timeout:${agentTimeout}`,
`maxSockets:${agentMaxSockets}`,
].join(':')
if (opts.agent != null) { // `agent: false` has special behavior!
return opts.agent
}
// keep alive in AWS lambda makes no sense
const lambdaAgent = !isLambda ? null
: isHttps ? require('https').globalAgent
: require('http').globalAgent
if (isLambda && !pxuri) {
return lambdaAgent
}
if (AGENT_CACHE.peek(key)) {
return AGENT_CACHE.get(key)
}
if (pxuri) {
const pxopts = isLambda ? {
...opts,
agent: lambdaAgent,
} : opts
const proxy = getProxy(pxuri, pxopts, isHttps)
AGENT_CACHE.set(key, proxy)
return proxy
}
const agent = isHttps ? new HttpsAgent({
maxSockets: agentMaxSockets,
ca: opts.ca,
cert: opts.cert,
key: opts.key,
localAddress: opts.localAddress,
rejectUnauthorized: opts.rejectUnauthorized,
timeout: agentTimeout,
freeSocketTimeout: 15000,
lookup: dns.getLookup(opts.dns),
}) : new HttpAgent({
maxSockets: agentMaxSockets,
localAddress: opts.localAddress,
timeout: agentTimeout,
freeSocketTimeout: 15000,
lookup: dns.getLookup(opts.dns),
})
AGENT_CACHE.set(key, agent)
return agent
}
function checkNoProxy (uri, opts) {
const host = new url.URL(uri).hostname.split('.').reverse()
let noproxy = (opts.noProxy || getProcessEnv('no_proxy'))
if (typeof noproxy === 'string') {
noproxy = noproxy.split(',').map(n => n.trim())
}
return noproxy && noproxy.some(no => {
const noParts = no.split('.').filter(x => x).reverse()
if (!noParts.length) {
return false
}
for (let i = 0; i < noParts.length; i++) {
if (host[i] !== noParts[i]) {
return false
}
}
return true
})
}
module.exports.getProcessEnv = getProcessEnv
function getProcessEnv (env) {
if (!env) {
return
}
let value
if (Array.isArray(env)) {
for (const e of env) {
value = process.env[e] ||
process.env[e.toUpperCase()] ||
process.env[e.toLowerCase()]
if (typeof value !== 'undefined') {
break
}
}
}
if (typeof env === 'string') {
value = process.env[env] ||
process.env[env.toUpperCase()] ||
process.env[env.toLowerCase()]
}
return value
}
module.exports.getProxyUri = getProxyUri
function getProxyUri (uri, opts) {
const protocol = new url.URL(uri).protocol
const proxy = opts.proxy ||
(
protocol === 'https:' &&
getProcessEnv('https_proxy')
) ||
(
protocol === 'http:' &&
getProcessEnv(['https_proxy', 'http_proxy', 'proxy'])
)
if (!proxy) {
return null
}
const parsedProxy = (typeof proxy === 'string') ? new url.URL(proxy) : proxy
return !checkNoProxy(uri, opts) && parsedProxy
}
const getAuth = u =>
u.username && u.password ? decodeURIComponent(`${u.username}:${u.password}`)
: u.username ? decodeURIComponent(u.username)
: null
const getPath = u => u.pathname + u.search + u.hash
const HttpProxyAgent = require('http-proxy-agent')
const HttpsProxyAgent = require('https-proxy-agent')
const { SocksProxyAgent } = require('socks-proxy-agent')
module.exports.getProxy = getProxy
function getProxy (proxyUrl, opts, isHttps) {
// our current proxy agents do not support an overridden dns lookup method, so will not
// benefit from the dns cache
const popts = {
host: proxyUrl.hostname,
port: proxyUrl.port,
protocol: proxyUrl.protocol,
path: getPath(proxyUrl),
auth: getAuth(proxyUrl),
ca: opts.ca,
cert: opts.cert,
key: opts.key,
timeout: getAgentTimeout(opts.timeout),
localAddress: opts.localAddress,
maxSockets: getMaxSockets(opts.maxSockets),
rejectUnauthorized: opts.rejectUnauthorized,
}
if (proxyUrl.protocol === 'http:' || proxyUrl.protocol === 'https:') {
if (!isHttps) {
return new HttpProxyAgent(popts)
} else {
return new HttpsProxyAgent(popts)
}
} else if (proxyUrl.protocol.startsWith('socks')) {
// socks-proxy-agent uses hostname not host
popts.hostname = popts.host
delete popts.host
return new SocksProxyAgent(popts)
} else {
throw Object.assign(
new Error(`unsupported proxy protocol: '${proxyUrl.protocol}'`),
{
code: 'EUNSUPPORTEDPROXY',
url: proxyUrl.href,
}
)
}
}

View File

@@ -0,0 +1,444 @@
const { Request, Response } = require('minipass-fetch')
const Minipass = require('minipass')
const MinipassFlush = require('minipass-flush')
const cacache = require('cacache')
const url = require('url')
const CachingMinipassPipeline = require('../pipeline.js')
const CachePolicy = require('./policy.js')
const cacheKey = require('./key.js')
const remote = require('../remote.js')
const hasOwnProperty = (obj, prop) => Object.prototype.hasOwnProperty.call(obj, prop)
// allow list for request headers that will be written to the cache index
// note: we will also store any request headers
// that are named in a response's vary header
const KEEP_REQUEST_HEADERS = [
'accept-charset',
'accept-encoding',
'accept-language',
'accept',
'cache-control',
]
// allow list for response headers that will be written to the cache index
// note: we must not store the real response's age header, or when we load
// a cache policy based on the metadata it will think the cached response
// is always stale
const KEEP_RESPONSE_HEADERS = [
'cache-control',
'content-encoding',
'content-language',
'content-type',
'date',
'etag',
'expires',
'last-modified',
'link',
'location',
'pragma',
'vary',
]
// return an object containing all metadata to be written to the index
const getMetadata = (request, response, options) => {
const metadata = {
time: Date.now(),
url: request.url,
reqHeaders: {},
resHeaders: {},
// options on which we must match the request and vary the response
options: {
compress: options.compress != null ? options.compress : request.compress,
},
}
// only save the status if it's not a 200 or 304
if (response.status !== 200 && response.status !== 304) {
metadata.status = response.status
}
for (const name of KEEP_REQUEST_HEADERS) {
if (request.headers.has(name)) {
metadata.reqHeaders[name] = request.headers.get(name)
}
}
// if the request's host header differs from the host in the url
// we need to keep it, otherwise it's just noise and we ignore it
const host = request.headers.get('host')
const parsedUrl = new url.URL(request.url)
if (host && parsedUrl.host !== host) {
metadata.reqHeaders.host = host
}
// if the response has a vary header, make sure
// we store the relevant request headers too
if (response.headers.has('vary')) {
const vary = response.headers.get('vary')
// a vary of "*" means every header causes a different response.
// in that scenario, we do not include any additional headers
// as the freshness check will always fail anyway and we don't
// want to bloat the cache indexes
if (vary !== '*') {
// copy any other request headers that will vary the response
const varyHeaders = vary.trim().toLowerCase().split(/\s*,\s*/)
for (const name of varyHeaders) {
if (request.headers.has(name)) {
metadata.reqHeaders[name] = request.headers.get(name)
}
}
}
}
for (const name of KEEP_RESPONSE_HEADERS) {
if (response.headers.has(name)) {
metadata.resHeaders[name] = response.headers.get(name)
}
}
return metadata
}
// symbols used to hide objects that may be lazily evaluated in a getter
const _request = Symbol('request')
const _response = Symbol('response')
const _policy = Symbol('policy')
class CacheEntry {
constructor ({ entry, request, response, options }) {
if (entry) {
this.key = entry.key
this.entry = entry
// previous versions of this module didn't write an explicit timestamp in
// the metadata, so fall back to the entry's timestamp. we can't use the
// entry timestamp to determine staleness because cacache will update it
// when it verifies its data
this.entry.metadata.time = this.entry.metadata.time || this.entry.time
} else {
this.key = cacheKey(request)
}
this.options = options
// these properties are behind getters that lazily evaluate
this[_request] = request
this[_response] = response
this[_policy] = null
}
// returns a CacheEntry instance that satisfies the given request
// or undefined if no existing entry satisfies
static async find (request, options) {
try {
// compacts the index and returns an array of unique entries
var matches = await cacache.index.compact(options.cachePath, cacheKey(request), (A, B) => {
const entryA = new CacheEntry({ entry: A, options })
const entryB = new CacheEntry({ entry: B, options })
return entryA.policy.satisfies(entryB.request)
}, {
validateEntry: (entry) => {
// clean out entries with a buggy content-encoding value
if (entry.metadata &&
entry.metadata.resHeaders &&
entry.metadata.resHeaders['content-encoding'] === null) {
return false
}
// if an integrity is null, it needs to have a status specified
if (entry.integrity === null) {
return !!(entry.metadata && entry.metadata.status)
}
return true
},
})
} catch (err) {
// if the compact request fails, ignore the error and return
return
}
// a cache mode of 'reload' means to behave as though we have no cache
// on the way to the network. return undefined to allow cacheFetch to
// create a brand new request no matter what.
if (options.cache === 'reload') {
return
}
// find the specific entry that satisfies the request
let match
for (const entry of matches) {
const _entry = new CacheEntry({
entry,
options,
})
if (_entry.policy.satisfies(request)) {
match = _entry
break
}
}
return match
}
// if the user made a PUT/POST/PATCH then we invalidate our
// cache for the same url by deleting the index entirely
static async invalidate (request, options) {
const key = cacheKey(request)
try {
await cacache.rm.entry(options.cachePath, key, { removeFully: true })
} catch (err) {
// ignore errors
}
}
get request () {
if (!this[_request]) {
this[_request] = new Request(this.entry.metadata.url, {
method: 'GET',
headers: this.entry.metadata.reqHeaders,
...this.entry.metadata.options,
})
}
return this[_request]
}
get response () {
if (!this[_response]) {
this[_response] = new Response(null, {
url: this.entry.metadata.url,
counter: this.options.counter,
status: this.entry.metadata.status || 200,
headers: {
...this.entry.metadata.resHeaders,
'content-length': this.entry.size,
},
})
}
return this[_response]
}
get policy () {
if (!this[_policy]) {
this[_policy] = new CachePolicy({
entry: this.entry,
request: this.request,
response: this.response,
options: this.options,
})
}
return this[_policy]
}
// wraps the response in a pipeline that stores the data
// in the cache while the user consumes it
async store (status) {
// if we got a status other than 200, 301, or 308,
// or the CachePolicy forbid storage, append the
// cache status header and return it untouched
if (
this.request.method !== 'GET' ||
![200, 301, 308].includes(this.response.status) ||
!this.policy.storable()
) {
this.response.headers.set('x-local-cache-status', 'skip')
return this.response
}
const size = this.response.headers.get('content-length')
const cacheOpts = {
algorithms: this.options.algorithms,
metadata: getMetadata(this.request, this.response, this.options),
size,
integrity: this.options.integrity,
integrityEmitter: this.response.body.hasIntegrityEmitter && this.response.body,
}
let body = null
// we only set a body if the status is a 200, redirects are
// stored as metadata only
if (this.response.status === 200) {
let cacheWriteResolve, cacheWriteReject
const cacheWritePromise = new Promise((resolve, reject) => {
cacheWriteResolve = resolve
cacheWriteReject = reject
})
body = new CachingMinipassPipeline({ events: ['integrity', 'size'] }, new MinipassFlush({
flush () {
return cacheWritePromise
},
}))
// this is always true since if we aren't reusing the one from the remote fetch, we
// are using the one from cacache
body.hasIntegrityEmitter = true
const onResume = () => {
const tee = new Minipass()
const cacheStream = cacache.put.stream(this.options.cachePath, this.key, cacheOpts)
// re-emit the integrity and size events on our new response body so they can be reused
cacheStream.on('integrity', i => body.emit('integrity', i))
cacheStream.on('size', s => body.emit('size', s))
// stick a flag on here so downstream users will know if they can expect integrity events
tee.pipe(cacheStream)
// TODO if the cache write fails, log a warning but return the response anyway
// eslint-disable-next-line promise/catch-or-return
cacheStream.promise().then(cacheWriteResolve, cacheWriteReject)
body.unshift(tee)
body.unshift(this.response.body)
}
body.once('resume', onResume)
body.once('end', () => body.removeListener('resume', onResume))
} else {
await cacache.index.insert(this.options.cachePath, this.key, null, cacheOpts)
}
// note: we do not set the x-local-cache-hash header because we do not know
// the hash value until after the write to the cache completes, which doesn't
// happen until after the response has been sent and it's too late to write
// the header anyway
this.response.headers.set('x-local-cache', encodeURIComponent(this.options.cachePath))
this.response.headers.set('x-local-cache-key', encodeURIComponent(this.key))
this.response.headers.set('x-local-cache-mode', 'stream')
this.response.headers.set('x-local-cache-status', status)
this.response.headers.set('x-local-cache-time', new Date().toISOString())
const newResponse = new Response(body, {
url: this.response.url,
status: this.response.status,
headers: this.response.headers,
counter: this.options.counter,
})
return newResponse
}
// use the cached data to create a response and return it
async respond (method, options, status) {
let response
if (method === 'HEAD' || [301, 308].includes(this.response.status)) {
// if the request is a HEAD, or the response is a redirect,
// then the metadata in the entry already includes everything
// we need to build a response
response = this.response
} else {
// we're responding with a full cached response, so create a body
// that reads from cacache and attach it to a new Response
const body = new Minipass()
const headers = { ...this.policy.responseHeaders() }
const onResume = () => {
const cacheStream = cacache.get.stream.byDigest(
this.options.cachePath, this.entry.integrity, { memoize: this.options.memoize }
)
cacheStream.on('error', async (err) => {
cacheStream.pause()
if (err.code === 'EINTEGRITY') {
await cacache.rm.content(
this.options.cachePath, this.entry.integrity, { memoize: this.options.memoize }
)
}
if (err.code === 'ENOENT' || err.code === 'EINTEGRITY') {
await CacheEntry.invalidate(this.request, this.options)
}
body.emit('error', err)
cacheStream.resume()
})
// emit the integrity and size events based on our metadata so we're consistent
body.emit('integrity', this.entry.integrity)
body.emit('size', Number(headers['content-length']))
cacheStream.pipe(body)
}
body.once('resume', onResume)
body.once('end', () => body.removeListener('resume', onResume))
response = new Response(body, {
url: this.entry.metadata.url,
counter: options.counter,
status: 200,
headers,
})
}
response.headers.set('x-local-cache', encodeURIComponent(this.options.cachePath))
response.headers.set('x-local-cache-hash', encodeURIComponent(this.entry.integrity))
response.headers.set('x-local-cache-key', encodeURIComponent(this.key))
response.headers.set('x-local-cache-mode', 'stream')
response.headers.set('x-local-cache-status', status)
response.headers.set('x-local-cache-time', new Date(this.entry.metadata.time).toUTCString())
return response
}
// use the provided request along with this cache entry to
// revalidate the stored response. returns a response, either
// from the cache or from the update
async revalidate (request, options) {
const revalidateRequest = new Request(request, {
headers: this.policy.revalidationHeaders(request),
})
try {
// NOTE: be sure to remove the headers property from the
// user supplied options, since we have already defined
// them on the new request object. if they're still in the
// options then those will overwrite the ones from the policy
var response = await remote(revalidateRequest, {
...options,
headers: undefined,
})
} catch (err) {
// if the network fetch fails, return the stale
// cached response unless it has a cache-control
// of 'must-revalidate'
if (!this.policy.mustRevalidate) {
return this.respond(request.method, options, 'stale')
}
throw err
}
if (this.policy.revalidated(revalidateRequest, response)) {
// we got a 304, write a new index to the cache and respond from cache
const metadata = getMetadata(request, response, options)
// 304 responses do not include headers that are specific to the response data
// since they do not include a body, so we copy values for headers that were
// in the old cache entry to the new one, if the new metadata does not already
// include that header
for (const name of KEEP_RESPONSE_HEADERS) {
if (
!hasOwnProperty(metadata.resHeaders, name) &&
hasOwnProperty(this.entry.metadata.resHeaders, name)
) {
metadata.resHeaders[name] = this.entry.metadata.resHeaders[name]
}
}
try {
await cacache.index.insert(options.cachePath, this.key, this.entry.integrity, {
size: this.entry.size,
metadata,
})
} catch (err) {
// if updating the cache index fails, we ignore it and
// respond anyway
}
return this.respond(request.method, options, 'revalidated')
}
// if we got a modified response, create a new entry based on it
const newEntry = new CacheEntry({
request,
response,
options,
})
// respond with the new entry while writing it to the cache
return newEntry.store('updated')
}
}
module.exports = CacheEntry

View File

@@ -0,0 +1,11 @@
class NotCachedError extends Error {
constructor (url) {
/* eslint-disable-next-line max-len */
super(`request to ${url} failed: cache mode is 'only-if-cached' but no cached response is available.`)
this.code = 'ENOTCACHED'
}
}
module.exports = {
NotCachedError,
}

View File

@@ -0,0 +1,49 @@
const { NotCachedError } = require('./errors.js')
const CacheEntry = require('./entry.js')
const remote = require('../remote.js')
// do whatever is necessary to get a Response and return it
const cacheFetch = async (request, options) => {
// try to find a cached entry that satisfies this request
const entry = await CacheEntry.find(request, options)
if (!entry) {
// no cached result, if the cache mode is 'only-if-cached' that's a failure
if (options.cache === 'only-if-cached') {
throw new NotCachedError(request.url)
}
// otherwise, we make a request, store it and return it
const response = await remote(request, options)
const newEntry = new CacheEntry({ request, response, options })
return newEntry.store('miss')
}
// we have a cached response that satisfies this request, however if the cache
// mode is 'no-cache' then we send the revalidation request no matter what
if (options.cache === 'no-cache') {
return entry.revalidate(request, options)
}
// if the cached entry is not stale, or if the cache mode is 'force-cache' or
// 'only-if-cached' we can respond with the cached entry. set the status
// based on the result of needsRevalidation and respond
const _needsRevalidation = entry.policy.needsRevalidation(request)
if (options.cache === 'force-cache' ||
options.cache === 'only-if-cached' ||
!_needsRevalidation) {
return entry.respond(request.method, options, _needsRevalidation ? 'stale' : 'hit')
}
// if we got here, the cache entry is stale so revalidate it
return entry.revalidate(request, options)
}
cacheFetch.invalidate = async (request, options) => {
if (!options.cachePath) {
return
}
return CacheEntry.invalidate(request, options)
}
module.exports = cacheFetch

View File

@@ -0,0 +1,17 @@
const { URL, format } = require('url')
// options passed to url.format() when generating a key
const formatOptions = {
auth: false,
fragment: false,
search: true,
unicode: false,
}
// returns a string to be used as the cache key for the Request
const cacheKey = (request) => {
const parsed = new URL(request.url)
return `make-fetch-happen:request-cache:${format(parsed, formatOptions)}`
}
module.exports = cacheKey

View File

@@ -0,0 +1,161 @@
const CacheSemantics = require('http-cache-semantics')
const Negotiator = require('negotiator')
const ssri = require('ssri')
// options passed to http-cache-semantics constructor
const policyOptions = {
shared: false,
ignoreCargoCult: true,
}
// a fake empty response, used when only testing the
// request for storability
const emptyResponse = { status: 200, headers: {} }
// returns a plain object representation of the Request
const requestObject = (request) => {
const _obj = {
method: request.method,
url: request.url,
headers: {},
compress: request.compress,
}
request.headers.forEach((value, key) => {
_obj.headers[key] = value
})
return _obj
}
// returns a plain object representation of the Response
const responseObject = (response) => {
const _obj = {
status: response.status,
headers: {},
}
response.headers.forEach((value, key) => {
_obj.headers[key] = value
})
return _obj
}
class CachePolicy {
constructor ({ entry, request, response, options }) {
this.entry = entry
this.request = requestObject(request)
this.response = responseObject(response)
this.options = options
this.policy = new CacheSemantics(this.request, this.response, policyOptions)
if (this.entry) {
// if we have an entry, copy the timestamp to the _responseTime
// this is necessary because the CacheSemantics constructor forces
// the value to Date.now() which means a policy created from a
// cache entry is likely to always identify itself as stale
this.policy._responseTime = this.entry.metadata.time
}
}
// static method to quickly determine if a request alone is storable
static storable (request, options) {
// no cachePath means no caching
if (!options.cachePath) {
return false
}
// user explicitly asked not to cache
if (options.cache === 'no-store') {
return false
}
// we only cache GET and HEAD requests
if (!['GET', 'HEAD'].includes(request.method)) {
return false
}
// otherwise, let http-cache-semantics make the decision
// based on the request's headers
const policy = new CacheSemantics(requestObject(request), emptyResponse, policyOptions)
return policy.storable()
}
// returns true if the policy satisfies the request
satisfies (request) {
const _req = requestObject(request)
if (this.request.headers.host !== _req.headers.host) {
return false
}
if (this.request.compress !== _req.compress) {
return false
}
const negotiatorA = new Negotiator(this.request)
const negotiatorB = new Negotiator(_req)
if (JSON.stringify(negotiatorA.mediaTypes()) !== JSON.stringify(negotiatorB.mediaTypes())) {
return false
}
if (JSON.stringify(negotiatorA.languages()) !== JSON.stringify(negotiatorB.languages())) {
return false
}
if (JSON.stringify(negotiatorA.encodings()) !== JSON.stringify(negotiatorB.encodings())) {
return false
}
if (this.options.integrity) {
return ssri.parse(this.options.integrity).match(this.entry.integrity)
}
return true
}
// returns true if the request and response allow caching
storable () {
return this.policy.storable()
}
// NOTE: this is a hack to avoid parsing the cache-control
// header ourselves, it returns true if the response's
// cache-control contains must-revalidate
get mustRevalidate () {
return !!this.policy._rescc['must-revalidate']
}
// returns true if the cached response requires revalidation
// for the given request
needsRevalidation (request) {
const _req = requestObject(request)
// force method to GET because we only cache GETs
// but can serve a HEAD from a cached GET
_req.method = 'GET'
return !this.policy.satisfiesWithoutRevalidation(_req)
}
responseHeaders () {
return this.policy.responseHeaders()
}
// returns a new object containing the appropriate headers
// to send a revalidation request
revalidationHeaders (request) {
const _req = requestObject(request)
return this.policy.revalidationHeaders(_req)
}
// returns true if the request/response was revalidated
// successfully. returns false if a new response was received
revalidated (request, response) {
const _req = requestObject(request)
const _res = responseObject(response)
const policy = this.policy.revalidatedPolicy(_req, _res)
return !policy.modified
}
}
module.exports = CachePolicy

View File

@@ -0,0 +1,49 @@
const LRUCache = require('lru-cache')
const dns = require('dns')
const defaultOptions = exports.defaultOptions = {
family: undefined,
hints: dns.ADDRCONFIG,
all: false,
verbatim: undefined,
}
const lookupCache = exports.lookupCache = new LRUCache({ max: 50 })
// this is a factory so that each request can have its own opts (i.e. ttl)
// while still sharing the cache across all requests
exports.getLookup = (dnsOptions) => {
return (hostname, options, callback) => {
if (typeof options === 'function') {
callback = options
options = null
} else if (typeof options === 'number') {
options = { family: options }
}
options = { ...defaultOptions, ...options }
const key = JSON.stringify({
hostname,
family: options.family,
hints: options.hints,
all: options.all,
verbatim: options.verbatim,
})
if (lookupCache.has(key)) {
const [address, family] = lookupCache.get(key)
process.nextTick(callback, null, address, family)
return
}
dnsOptions.lookup(hostname, options, (err, address, family) => {
if (err) {
return callback(err)
}
lookupCache.set(key, [address, family], { ttl: dnsOptions.ttl })
return callback(null, address, family)
})
}
}

View File

@@ -0,0 +1,118 @@
'use strict'
const { FetchError, Request, isRedirect } = require('minipass-fetch')
const url = require('url')
const CachePolicy = require('./cache/policy.js')
const cache = require('./cache/index.js')
const remote = require('./remote.js')
// given a Request, a Response and user options
// return true if the response is a redirect that
// can be followed. we throw errors that will result
// in the fetch being rejected if the redirect is
// possible but invalid for some reason
const canFollowRedirect = (request, response, options) => {
if (!isRedirect(response.status)) {
return false
}
if (options.redirect === 'manual') {
return false
}
if (options.redirect === 'error') {
throw new FetchError(`redirect mode is set to error: ${request.url}`,
'no-redirect', { code: 'ENOREDIRECT' })
}
if (!response.headers.has('location')) {
throw new FetchError(`redirect location header missing for: ${request.url}`,
'no-location', { code: 'EINVALIDREDIRECT' })
}
if (request.counter >= request.follow) {
throw new FetchError(`maximum redirect reached at: ${request.url}`,
'max-redirect', { code: 'EMAXREDIRECT' })
}
return true
}
// given a Request, a Response, and the user's options return an object
// with a new Request and a new options object that will be used for
// following the redirect
const getRedirect = (request, response, options) => {
const _opts = { ...options }
const location = response.headers.get('location')
const redirectUrl = new url.URL(location, /^https?:/.test(location) ? undefined : request.url)
// Comment below is used under the following license:
/**
* @license
* Copyright (c) 2010-2012 Mikeal Rogers
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an "AS
* IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
* express or implied. See the License for the specific language
* governing permissions and limitations under the License.
*/
// Remove authorization if changing hostnames (but not if just
// changing ports or protocols). This matches the behavior of request:
// https://github.com/request/request/blob/b12a6245/lib/redirect.js#L134-L138
if (new url.URL(request.url).hostname !== redirectUrl.hostname) {
request.headers.delete('authorization')
request.headers.delete('cookie')
}
// for POST request with 301/302 response, or any request with 303 response,
// use GET when following redirect
if (
response.status === 303 ||
(request.method === 'POST' && [301, 302].includes(response.status))
) {
_opts.method = 'GET'
_opts.body = null
request.headers.delete('content-length')
}
_opts.headers = {}
request.headers.forEach((value, key) => {
_opts.headers[key] = value
})
_opts.counter = ++request.counter
const redirectReq = new Request(url.format(redirectUrl), _opts)
return {
request: redirectReq,
options: _opts,
}
}
const fetch = async (request, options) => {
const response = CachePolicy.storable(request, options)
? await cache(request, options)
: await remote(request, options)
// if the request wasn't a GET or HEAD, and the response
// status is between 200 and 399 inclusive, invalidate the
// request url
if (!['GET', 'HEAD'].includes(request.method) &&
response.status >= 200 &&
response.status <= 399) {
await cache.invalidate(request, options)
}
if (!canFollowRedirect(request, response, options)) {
return response
}
const redirect = getRedirect(request, response, options)
return fetch(redirect.request, redirect.options)
}
module.exports = fetch

View File

@@ -0,0 +1,41 @@
const { FetchError, Headers, Request, Response } = require('minipass-fetch')
const configureOptions = require('./options.js')
const fetch = require('./fetch.js')
const makeFetchHappen = (url, opts) => {
const options = configureOptions(opts)
const request = new Request(url, options)
return fetch(request, options)
}
makeFetchHappen.defaults = (defaultUrl, defaultOptions = {}, wrappedFetch = makeFetchHappen) => {
if (typeof defaultUrl === 'object') {
defaultOptions = defaultUrl
defaultUrl = null
}
const defaultedFetch = (url, options = {}) => {
const finalUrl = url || defaultUrl
const finalOptions = {
...defaultOptions,
...options,
headers: {
...defaultOptions.headers,
...options.headers,
},
}
return wrappedFetch(finalUrl, finalOptions)
}
defaultedFetch.defaults = (defaultUrl1, defaultOptions1 = {}) =>
makeFetchHappen.defaults(defaultUrl1, defaultOptions1, defaultedFetch)
return defaultedFetch
}
module.exports = makeFetchHappen
module.exports.FetchError = FetchError
module.exports.Headers = Headers
module.exports.Request = Request
module.exports.Response = Response

View File

@@ -0,0 +1,52 @@
const dns = require('dns')
const conditionalHeaders = [
'if-modified-since',
'if-none-match',
'if-unmodified-since',
'if-match',
'if-range',
]
const configureOptions = (opts) => {
const { strictSSL, ...options } = { ...opts }
options.method = options.method ? options.method.toUpperCase() : 'GET'
options.rejectUnauthorized = strictSSL !== false
if (!options.retry) {
options.retry = { retries: 0 }
} else if (typeof options.retry === 'string') {
const retries = parseInt(options.retry, 10)
if (isFinite(retries)) {
options.retry = { retries }
} else {
options.retry = { retries: 0 }
}
} else if (typeof options.retry === 'number') {
options.retry = { retries: options.retry }
} else {
options.retry = { retries: 0, ...options.retry }
}
options.dns = { ttl: 5 * 60 * 1000, lookup: dns.lookup, ...options.dns }
options.cache = options.cache || 'default'
if (options.cache === 'default') {
const hasConditionalHeader = Object.keys(options.headers || {}).some((name) => {
return conditionalHeaders.includes(name.toLowerCase())
})
if (hasConditionalHeader) {
options.cache = 'no-store'
}
}
// cacheManager is deprecated, but if it's set and
// cachePath is not we should copy it to the new field
if (options.cacheManager && !options.cachePath) {
options.cachePath = options.cacheManager
}
return options
}
module.exports = configureOptions

View File

@@ -0,0 +1,41 @@
'use strict'
const MinipassPipeline = require('minipass-pipeline')
class CachingMinipassPipeline extends MinipassPipeline {
#events = []
#data = new Map()
constructor (opts, ...streams) {
// CRITICAL: do NOT pass the streams to the call to super(), this will start
// the flow of data and potentially cause the events we need to catch to emit
// before we've finished our own setup. instead we call super() with no args,
// finish our setup, and then push the streams into ourselves to start the
// data flow
super()
this.#events = opts.events
/* istanbul ignore next - coverage disabled because this is pointless to test here */
if (streams.length) {
this.push(...streams)
}
}
on (event, handler) {
if (this.#events.includes(event) && this.#data.has(event)) {
return handler(...this.#data.get(event))
}
return super.on(event, handler)
}
emit (event, ...data) {
if (this.#events.includes(event)) {
this.#data.set(event, data)
}
return super.emit(event, ...data)
}
}
module.exports = CachingMinipassPipeline

View File

@@ -0,0 +1,121 @@
const Minipass = require('minipass')
const fetch = require('minipass-fetch')
const promiseRetry = require('promise-retry')
const ssri = require('ssri')
const CachingMinipassPipeline = require('./pipeline.js')
const getAgent = require('./agent.js')
const pkg = require('../package.json')
const USER_AGENT = `${pkg.name}/${pkg.version} (+https://npm.im/${pkg.name})`
const RETRY_ERRORS = [
'ECONNRESET', // remote socket closed on us
'ECONNREFUSED', // remote host refused to open connection
'EADDRINUSE', // failed to bind to a local port (proxy?)
'ETIMEDOUT', // someone in the transaction is WAY TOO SLOW
'ERR_SOCKET_TIMEOUT', // same as above, but this one comes from agentkeepalive
// Known codes we do NOT retry on:
// ENOTFOUND (getaddrinfo failure. Either bad hostname, or offline)
]
const RETRY_TYPES = [
'request-timeout',
]
// make a request directly to the remote source,
// retrying certain classes of errors as well as
// following redirects (through the cache if necessary)
// and verifying response integrity
const remoteFetch = (request, options) => {
const agent = getAgent(request.url, options)
if (!request.headers.has('connection')) {
request.headers.set('connection', agent ? 'keep-alive' : 'close')
}
if (!request.headers.has('user-agent')) {
request.headers.set('user-agent', USER_AGENT)
}
// keep our own options since we're overriding the agent
// and the redirect mode
const _opts = {
...options,
agent,
redirect: 'manual',
}
return promiseRetry(async (retryHandler, attemptNum) => {
const req = new fetch.Request(request, _opts)
try {
let res = await fetch(req, _opts)
if (_opts.integrity && res.status === 200) {
// we got a 200 response and the user has specified an expected
// integrity value, so wrap the response in an ssri stream to verify it
const integrityStream = ssri.integrityStream({
algorithms: _opts.algorithms,
integrity: _opts.integrity,
size: _opts.size,
})
const pipeline = new CachingMinipassPipeline({
events: ['integrity', 'size'],
}, res.body, integrityStream)
// we also propagate the integrity and size events out to the pipeline so we can use
// this new response body as an integrityEmitter for cacache
integrityStream.on('integrity', i => pipeline.emit('integrity', i))
integrityStream.on('size', s => pipeline.emit('size', s))
res = new fetch.Response(pipeline, res)
// set an explicit flag so we know if our response body will emit integrity and size
res.body.hasIntegrityEmitter = true
}
res.headers.set('x-fetch-attempts', attemptNum)
// do not retry POST requests, or requests with a streaming body
// do retry requests with a 408, 420, 429 or 500+ status in the response
const isStream = Minipass.isStream(req.body)
const isRetriable = req.method !== 'POST' &&
!isStream &&
([408, 420, 429].includes(res.status) || res.status >= 500)
if (isRetriable) {
if (typeof options.onRetry === 'function') {
options.onRetry(res)
}
return retryHandler(res)
}
return res
} catch (err) {
const code = (err.code === 'EPROMISERETRY')
? err.retried.code
: err.code
// err.retried will be the thing that was thrown from above
// if it's a response, we just got a bad status code and we
// can re-throw to allow the retry
const isRetryError = err.retried instanceof fetch.Response ||
(RETRY_ERRORS.includes(code) && RETRY_TYPES.includes(err.type))
if (req.method === 'POST' || isRetryError) {
throw err
}
if (typeof options.onRetry === 'function') {
options.onRetry(err)
}
return retryHandler(err)
}
}, options.retry).catch((err) => {
// don't reject for http errors, just return them
if (err.status >= 400 && err.type !== 'system') {
return err
}
throw err
})
}
module.exports = remoteFetch

View File

@@ -0,0 +1,145 @@
agent-base
==========
### Turn a function into an [`http.Agent`][http.Agent] instance
[![Build Status](https://github.com/TooTallNate/node-agent-base/workflows/Node%20CI/badge.svg)](https://github.com/TooTallNate/node-agent-base/actions?workflow=Node+CI)
This module provides an `http.Agent` generator. That is, you pass it an async
callback function, and it returns a new `http.Agent` instance that will invoke the
given callback function when sending outbound HTTP requests.
#### Some subclasses:
Here's some more interesting uses of `agent-base`.
Send a pull request to list yours!
* [`http-proxy-agent`][http-proxy-agent]: An HTTP(s) proxy `http.Agent` implementation for HTTP endpoints
* [`https-proxy-agent`][https-proxy-agent]: An HTTP(s) proxy `http.Agent` implementation for HTTPS endpoints
* [`pac-proxy-agent`][pac-proxy-agent]: A PAC file proxy `http.Agent` implementation for HTTP and HTTPS
* [`socks-proxy-agent`][socks-proxy-agent]: A SOCKS proxy `http.Agent` implementation for HTTP and HTTPS
Installation
------------
Install with `npm`:
``` bash
$ npm install agent-base
```
Example
-------
Here's a minimal example that creates a new `net.Socket` connection to the server
for every HTTP request (i.e. the equivalent of `agent: false` option):
```js
var net = require('net');
var tls = require('tls');
var url = require('url');
var http = require('http');
var agent = require('agent-base');
var endpoint = 'http://nodejs.org/api/';
var parsed = url.parse(endpoint);
// This is the important part!
parsed.agent = agent(function (req, opts) {
var socket;
// `secureEndpoint` is true when using the https module
if (opts.secureEndpoint) {
socket = tls.connect(opts);
} else {
socket = net.connect(opts);
}
return socket;
});
// Everything else works just like normal...
http.get(parsed, function (res) {
console.log('"response" event!', res.headers);
res.pipe(process.stdout);
});
```
Returning a Promise or using an `async` function is also supported:
```js
agent(async function (req, opts) {
await sleep(1000);
// etc…
});
```
Return another `http.Agent` instance to "pass through" the responsibility
for that HTTP request to that agent:
```js
agent(function (req, opts) {
return opts.secureEndpoint ? https.globalAgent : http.globalAgent;
});
```
API
---
## Agent(Function callback[, Object options]) → [http.Agent][]
Creates a base `http.Agent` that will execute the callback function `callback`
for every HTTP request that it is used as the `agent` for. The callback function
is responsible for creating a `stream.Duplex` instance of some kind that will be
used as the underlying socket in the HTTP request.
The `options` object accepts the following properties:
* `timeout` - Number - Timeout for the `callback()` function in milliseconds. Defaults to Infinity (optional).
The callback function should have the following signature:
### callback(http.ClientRequest req, Object options, Function cb) → undefined
The ClientRequest `req` can be accessed to read request headers and
and the path, etc. The `options` object contains the options passed
to the `http.request()`/`https.request()` function call, and is formatted
to be directly passed to `net.connect()`/`tls.connect()`, or however
else you want a Socket to be created. Pass the created socket to
the callback function `cb` once created, and the HTTP request will
continue to proceed.
If the `https` module is used to invoke the HTTP request, then the
`secureEndpoint` property on `options` _will be set to `true`_.
License
-------
(The MIT License)
Copyright (c) 2013 Nathan Rajlich &lt;nathan@tootallnate.net&gt;
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
[http-proxy-agent]: https://github.com/TooTallNate/node-http-proxy-agent
[https-proxy-agent]: https://github.com/TooTallNate/node-https-proxy-agent
[pac-proxy-agent]: https://github.com/TooTallNate/node-pac-proxy-agent
[socks-proxy-agent]: https://github.com/TooTallNate/node-socks-proxy-agent
[http.Agent]: https://nodejs.org/api/http.html#http_class_http_agent

View File

@@ -0,0 +1,78 @@
/// <reference types="node" />
import net from 'net';
import http from 'http';
import https from 'https';
import { Duplex } from 'stream';
import { EventEmitter } from 'events';
declare function createAgent(opts?: createAgent.AgentOptions): createAgent.Agent;
declare function createAgent(callback: createAgent.AgentCallback, opts?: createAgent.AgentOptions): createAgent.Agent;
declare namespace createAgent {
interface ClientRequest extends http.ClientRequest {
_last?: boolean;
_hadError?: boolean;
method: string;
}
interface AgentRequestOptions {
host?: string;
path?: string;
port: number;
}
interface HttpRequestOptions extends AgentRequestOptions, Omit<http.RequestOptions, keyof AgentRequestOptions> {
secureEndpoint: false;
}
interface HttpsRequestOptions extends AgentRequestOptions, Omit<https.RequestOptions, keyof AgentRequestOptions> {
secureEndpoint: true;
}
type RequestOptions = HttpRequestOptions | HttpsRequestOptions;
type AgentLike = Pick<createAgent.Agent, 'addRequest'> | http.Agent;
type AgentCallbackReturn = Duplex | AgentLike;
type AgentCallbackCallback = (err?: Error | null, socket?: createAgent.AgentCallbackReturn) => void;
type AgentCallbackPromise = (req: createAgent.ClientRequest, opts: createAgent.RequestOptions) => createAgent.AgentCallbackReturn | Promise<createAgent.AgentCallbackReturn>;
type AgentCallback = typeof Agent.prototype.callback;
type AgentOptions = {
timeout?: number;
};
/**
* Base `http.Agent` implementation.
* No pooling/keep-alive is implemented by default.
*
* @param {Function} callback
* @api public
*/
class Agent extends EventEmitter {
timeout: number | null;
maxFreeSockets: number;
maxTotalSockets: number;
maxSockets: number;
sockets: {
[key: string]: net.Socket[];
};
freeSockets: {
[key: string]: net.Socket[];
};
requests: {
[key: string]: http.IncomingMessage[];
};
options: https.AgentOptions;
private promisifiedCallback?;
private explicitDefaultPort?;
private explicitProtocol?;
constructor(callback?: createAgent.AgentCallback | createAgent.AgentOptions, _opts?: createAgent.AgentOptions);
get defaultPort(): number;
set defaultPort(v: number);
get protocol(): string;
set protocol(v: string);
callback(req: createAgent.ClientRequest, opts: createAgent.RequestOptions, fn: createAgent.AgentCallbackCallback): void;
callback(req: createAgent.ClientRequest, opts: createAgent.RequestOptions): createAgent.AgentCallbackReturn | Promise<createAgent.AgentCallbackReturn>;
/**
* Called by node-core's "_http_client.js" module when creating
* a new HTTP request with this Agent instance.
*
* @api public
*/
addRequest(req: ClientRequest, _opts: RequestOptions): void;
freeSocket(socket: net.Socket, opts: AgentOptions): void;
destroy(): void;
}
}
export = createAgent;

View File

@@ -0,0 +1,203 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
const events_1 = require("events");
const debug_1 = __importDefault(require("debug"));
const promisify_1 = __importDefault(require("./promisify"));
const debug = debug_1.default('agent-base');
function isAgent(v) {
return Boolean(v) && typeof v.addRequest === 'function';
}
function isSecureEndpoint() {
const { stack } = new Error();
if (typeof stack !== 'string')
return false;
return stack.split('\n').some(l => l.indexOf('(https.js:') !== -1 || l.indexOf('node:https:') !== -1);
}
function createAgent(callback, opts) {
return new createAgent.Agent(callback, opts);
}
(function (createAgent) {
/**
* Base `http.Agent` implementation.
* No pooling/keep-alive is implemented by default.
*
* @param {Function} callback
* @api public
*/
class Agent extends events_1.EventEmitter {
constructor(callback, _opts) {
super();
let opts = _opts;
if (typeof callback === 'function') {
this.callback = callback;
}
else if (callback) {
opts = callback;
}
// Timeout for the socket to be returned from the callback
this.timeout = null;
if (opts && typeof opts.timeout === 'number') {
this.timeout = opts.timeout;
}
// These aren't actually used by `agent-base`, but are required
// for the TypeScript definition files in `@types/node` :/
this.maxFreeSockets = 1;
this.maxSockets = 1;
this.maxTotalSockets = Infinity;
this.sockets = {};
this.freeSockets = {};
this.requests = {};
this.options = {};
}
get defaultPort() {
if (typeof this.explicitDefaultPort === 'number') {
return this.explicitDefaultPort;
}
return isSecureEndpoint() ? 443 : 80;
}
set defaultPort(v) {
this.explicitDefaultPort = v;
}
get protocol() {
if (typeof this.explicitProtocol === 'string') {
return this.explicitProtocol;
}
return isSecureEndpoint() ? 'https:' : 'http:';
}
set protocol(v) {
this.explicitProtocol = v;
}
callback(req, opts, fn) {
throw new Error('"agent-base" has no default implementation, you must subclass and override `callback()`');
}
/**
* Called by node-core's "_http_client.js" module when creating
* a new HTTP request with this Agent instance.
*
* @api public
*/
addRequest(req, _opts) {
const opts = Object.assign({}, _opts);
if (typeof opts.secureEndpoint !== 'boolean') {
opts.secureEndpoint = isSecureEndpoint();
}
if (opts.host == null) {
opts.host = 'localhost';
}
if (opts.port == null) {
opts.port = opts.secureEndpoint ? 443 : 80;
}
if (opts.protocol == null) {
opts.protocol = opts.secureEndpoint ? 'https:' : 'http:';
}
if (opts.host && opts.path) {
// If both a `host` and `path` are specified then it's most
// likely the result of a `url.parse()` call... we need to
// remove the `path` portion so that `net.connect()` doesn't
// attempt to open that as a unix socket file.
delete opts.path;
}
delete opts.agent;
delete opts.hostname;
delete opts._defaultAgent;
delete opts.defaultPort;
delete opts.createConnection;
// Hint to use "Connection: close"
// XXX: non-documented `http` module API :(
req._last = true;
req.shouldKeepAlive = false;
let timedOut = false;
let timeoutId = null;
const timeoutMs = opts.timeout || this.timeout;
const onerror = (err) => {
if (req._hadError)
return;
req.emit('error', err);
// For Safety. Some additional errors might fire later on
// and we need to make sure we don't double-fire the error event.
req._hadError = true;
};
const ontimeout = () => {
timeoutId = null;
timedOut = true;
const err = new Error(`A "socket" was not created for HTTP request before ${timeoutMs}ms`);
err.code = 'ETIMEOUT';
onerror(err);
};
const callbackError = (err) => {
if (timedOut)
return;
if (timeoutId !== null) {
clearTimeout(timeoutId);
timeoutId = null;
}
onerror(err);
};
const onsocket = (socket) => {
if (timedOut)
return;
if (timeoutId != null) {
clearTimeout(timeoutId);
timeoutId = null;
}
if (isAgent(socket)) {
// `socket` is actually an `http.Agent` instance, so
// relinquish responsibility for this `req` to the Agent
// from here on
debug('Callback returned another Agent instance %o', socket.constructor.name);
socket.addRequest(req, opts);
return;
}
if (socket) {
socket.once('free', () => {
this.freeSocket(socket, opts);
});
req.onSocket(socket);
return;
}
const err = new Error(`no Duplex stream was returned to agent-base for \`${req.method} ${req.path}\``);
onerror(err);
};
if (typeof this.callback !== 'function') {
onerror(new Error('`callback` is not defined'));
return;
}
if (!this.promisifiedCallback) {
if (this.callback.length >= 3) {
debug('Converting legacy callback function to promise');
this.promisifiedCallback = promisify_1.default(this.callback);
}
else {
this.promisifiedCallback = this.callback;
}
}
if (typeof timeoutMs === 'number' && timeoutMs > 0) {
timeoutId = setTimeout(ontimeout, timeoutMs);
}
if ('port' in opts && typeof opts.port !== 'number') {
opts.port = Number(opts.port);
}
try {
debug('Resolving socket for %o request: %o', opts.protocol, `${req.method} ${req.path}`);
Promise.resolve(this.promisifiedCallback(req, opts)).then(onsocket, callbackError);
}
catch (err) {
Promise.reject(err).catch(callbackError);
}
}
freeSocket(socket, opts) {
debug('Freeing socket %o %o', socket.constructor.name, opts);
socket.destroy();
}
destroy() {
debug('Destroying agent %o', this.constructor.name);
}
}
createAgent.Agent = Agent;
// So that `instanceof` works correctly
createAgent.prototype = createAgent.Agent.prototype;
})(createAgent || (createAgent = {}));
module.exports = createAgent;
//# sourceMappingURL=index.js.map

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,4 @@
import { ClientRequest, RequestOptions, AgentCallbackCallback, AgentCallbackPromise } from './index';
declare type LegacyCallback = (req: ClientRequest, opts: RequestOptions, fn: AgentCallbackCallback) => void;
export default function promisify(fn: LegacyCallback): AgentCallbackPromise;
export {};

View File

@@ -0,0 +1,18 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
function promisify(fn) {
return function (req, opts) {
return new Promise((resolve, reject) => {
fn.call(this, req, opts, (err, rtn) => {
if (err) {
reject(err);
}
else {
resolve(rtn);
}
});
});
};
}
exports.default = promisify;
//# sourceMappingURL=promisify.js.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"promisify.js","sourceRoot":"","sources":["../../src/promisify.ts"],"names":[],"mappings":";;AAeA,SAAwB,SAAS,CAAC,EAAkB;IACnD,OAAO,UAAsB,GAAkB,EAAE,IAAoB;QACpE,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YACtC,EAAE,CAAC,IAAI,CACN,IAAI,EACJ,GAAG,EACH,IAAI,EACJ,CAAC,GAA6B,EAAE,GAAyB,EAAE,EAAE;gBAC5D,IAAI,GAAG,EAAE;oBACR,MAAM,CAAC,GAAG,CAAC,CAAC;iBACZ;qBAAM;oBACN,OAAO,CAAC,GAAG,CAAC,CAAC;iBACb;YACF,CAAC,CACD,CAAC;QACH,CAAC,CAAC,CAAC;IACJ,CAAC,CAAC;AACH,CAAC;AAjBD,4BAiBC"}

View File

@@ -0,0 +1,64 @@
{
"name": "agent-base",
"version": "6.0.2",
"description": "Turn a function into an `http.Agent` instance",
"main": "dist/src/index",
"typings": "dist/src/index",
"files": [
"dist/src",
"src"
],
"scripts": {
"prebuild": "rimraf dist",
"build": "tsc",
"postbuild": "cpy --parents src test '!**/*.ts' dist",
"test": "mocha --reporter spec dist/test/*.js",
"test-lint": "eslint src --ext .js,.ts",
"prepublishOnly": "npm run build"
},
"repository": {
"type": "git",
"url": "git://github.com/TooTallNate/node-agent-base.git"
},
"keywords": [
"http",
"agent",
"base",
"barebones",
"https"
],
"author": "Nathan Rajlich <nathan@tootallnate.net> (http://n8.io/)",
"license": "MIT",
"bugs": {
"url": "https://github.com/TooTallNate/node-agent-base/issues"
},
"dependencies": {
"debug": "4"
},
"devDependencies": {
"@types/debug": "4",
"@types/mocha": "^5.2.7",
"@types/node": "^14.0.20",
"@types/semver": "^7.1.0",
"@types/ws": "^6.0.3",
"@typescript-eslint/eslint-plugin": "1.6.0",
"@typescript-eslint/parser": "1.1.0",
"async-listen": "^1.2.0",
"cpy-cli": "^2.0.0",
"eslint": "5.16.0",
"eslint-config-airbnb": "17.1.0",
"eslint-config-prettier": "4.1.0",
"eslint-import-resolver-typescript": "1.1.1",
"eslint-plugin-import": "2.16.0",
"eslint-plugin-jsx-a11y": "6.2.1",
"eslint-plugin-react": "7.12.4",
"mocha": "^6.2.0",
"rimraf": "^3.0.0",
"semver": "^7.1.2",
"typescript": "^3.5.3",
"ws": "^3.0.0"
},
"engines": {
"node": ">= 6.0.0"
}
}

View File

@@ -0,0 +1,345 @@
import net from 'net';
import http from 'http';
import https from 'https';
import { Duplex } from 'stream';
import { EventEmitter } from 'events';
import createDebug from 'debug';
import promisify from './promisify';
const debug = createDebug('agent-base');
function isAgent(v: any): v is createAgent.AgentLike {
return Boolean(v) && typeof v.addRequest === 'function';
}
function isSecureEndpoint(): boolean {
const { stack } = new Error();
if (typeof stack !== 'string') return false;
return stack.split('\n').some(l => l.indexOf('(https.js:') !== -1 || l.indexOf('node:https:') !== -1);
}
function createAgent(opts?: createAgent.AgentOptions): createAgent.Agent;
function createAgent(
callback: createAgent.AgentCallback,
opts?: createAgent.AgentOptions
): createAgent.Agent;
function createAgent(
callback?: createAgent.AgentCallback | createAgent.AgentOptions,
opts?: createAgent.AgentOptions
) {
return new createAgent.Agent(callback, opts);
}
namespace createAgent {
export interface ClientRequest extends http.ClientRequest {
_last?: boolean;
_hadError?: boolean;
method: string;
}
export interface AgentRequestOptions {
host?: string;
path?: string;
// `port` on `http.RequestOptions` can be a string or undefined,
// but `net.TcpNetConnectOpts` expects only a number
port: number;
}
export interface HttpRequestOptions
extends AgentRequestOptions,
Omit<http.RequestOptions, keyof AgentRequestOptions> {
secureEndpoint: false;
}
export interface HttpsRequestOptions
extends AgentRequestOptions,
Omit<https.RequestOptions, keyof AgentRequestOptions> {
secureEndpoint: true;
}
export type RequestOptions = HttpRequestOptions | HttpsRequestOptions;
export type AgentLike = Pick<createAgent.Agent, 'addRequest'> | http.Agent;
export type AgentCallbackReturn = Duplex | AgentLike;
export type AgentCallbackCallback = (
err?: Error | null,
socket?: createAgent.AgentCallbackReturn
) => void;
export type AgentCallbackPromise = (
req: createAgent.ClientRequest,
opts: createAgent.RequestOptions
) =>
| createAgent.AgentCallbackReturn
| Promise<createAgent.AgentCallbackReturn>;
export type AgentCallback = typeof Agent.prototype.callback;
export type AgentOptions = {
timeout?: number;
};
/**
* Base `http.Agent` implementation.
* No pooling/keep-alive is implemented by default.
*
* @param {Function} callback
* @api public
*/
export class Agent extends EventEmitter {
public timeout: number | null;
public maxFreeSockets: number;
public maxTotalSockets: number;
public maxSockets: number;
public sockets: {
[key: string]: net.Socket[];
};
public freeSockets: {
[key: string]: net.Socket[];
};
public requests: {
[key: string]: http.IncomingMessage[];
};
public options: https.AgentOptions;
private promisifiedCallback?: createAgent.AgentCallbackPromise;
private explicitDefaultPort?: number;
private explicitProtocol?: string;
constructor(
callback?: createAgent.AgentCallback | createAgent.AgentOptions,
_opts?: createAgent.AgentOptions
) {
super();
let opts = _opts;
if (typeof callback === 'function') {
this.callback = callback;
} else if (callback) {
opts = callback;
}
// Timeout for the socket to be returned from the callback
this.timeout = null;
if (opts && typeof opts.timeout === 'number') {
this.timeout = opts.timeout;
}
// These aren't actually used by `agent-base`, but are required
// for the TypeScript definition files in `@types/node` :/
this.maxFreeSockets = 1;
this.maxSockets = 1;
this.maxTotalSockets = Infinity;
this.sockets = {};
this.freeSockets = {};
this.requests = {};
this.options = {};
}
get defaultPort(): number {
if (typeof this.explicitDefaultPort === 'number') {
return this.explicitDefaultPort;
}
return isSecureEndpoint() ? 443 : 80;
}
set defaultPort(v: number) {
this.explicitDefaultPort = v;
}
get protocol(): string {
if (typeof this.explicitProtocol === 'string') {
return this.explicitProtocol;
}
return isSecureEndpoint() ? 'https:' : 'http:';
}
set protocol(v: string) {
this.explicitProtocol = v;
}
callback(
req: createAgent.ClientRequest,
opts: createAgent.RequestOptions,
fn: createAgent.AgentCallbackCallback
): void;
callback(
req: createAgent.ClientRequest,
opts: createAgent.RequestOptions
):
| createAgent.AgentCallbackReturn
| Promise<createAgent.AgentCallbackReturn>;
callback(
req: createAgent.ClientRequest,
opts: createAgent.AgentOptions,
fn?: createAgent.AgentCallbackCallback
):
| createAgent.AgentCallbackReturn
| Promise<createAgent.AgentCallbackReturn>
| void {
throw new Error(
'"agent-base" has no default implementation, you must subclass and override `callback()`'
);
}
/**
* Called by node-core's "_http_client.js" module when creating
* a new HTTP request with this Agent instance.
*
* @api public
*/
addRequest(req: ClientRequest, _opts: RequestOptions): void {
const opts: RequestOptions = { ..._opts };
if (typeof opts.secureEndpoint !== 'boolean') {
opts.secureEndpoint = isSecureEndpoint();
}
if (opts.host == null) {
opts.host = 'localhost';
}
if (opts.port == null) {
opts.port = opts.secureEndpoint ? 443 : 80;
}
if (opts.protocol == null) {
opts.protocol = opts.secureEndpoint ? 'https:' : 'http:';
}
if (opts.host && opts.path) {
// If both a `host` and `path` are specified then it's most
// likely the result of a `url.parse()` call... we need to
// remove the `path` portion so that `net.connect()` doesn't
// attempt to open that as a unix socket file.
delete opts.path;
}
delete opts.agent;
delete opts.hostname;
delete opts._defaultAgent;
delete opts.defaultPort;
delete opts.createConnection;
// Hint to use "Connection: close"
// XXX: non-documented `http` module API :(
req._last = true;
req.shouldKeepAlive = false;
let timedOut = false;
let timeoutId: ReturnType<typeof setTimeout> | null = null;
const timeoutMs = opts.timeout || this.timeout;
const onerror = (err: NodeJS.ErrnoException) => {
if (req._hadError) return;
req.emit('error', err);
// For Safety. Some additional errors might fire later on
// and we need to make sure we don't double-fire the error event.
req._hadError = true;
};
const ontimeout = () => {
timeoutId = null;
timedOut = true;
const err: NodeJS.ErrnoException = new Error(
`A "socket" was not created for HTTP request before ${timeoutMs}ms`
);
err.code = 'ETIMEOUT';
onerror(err);
};
const callbackError = (err: NodeJS.ErrnoException) => {
if (timedOut) return;
if (timeoutId !== null) {
clearTimeout(timeoutId);
timeoutId = null;
}
onerror(err);
};
const onsocket = (socket: AgentCallbackReturn) => {
if (timedOut) return;
if (timeoutId != null) {
clearTimeout(timeoutId);
timeoutId = null;
}
if (isAgent(socket)) {
// `socket` is actually an `http.Agent` instance, so
// relinquish responsibility for this `req` to the Agent
// from here on
debug(
'Callback returned another Agent instance %o',
socket.constructor.name
);
(socket as createAgent.Agent).addRequest(req, opts);
return;
}
if (socket) {
socket.once('free', () => {
this.freeSocket(socket as net.Socket, opts);
});
req.onSocket(socket as net.Socket);
return;
}
const err = new Error(
`no Duplex stream was returned to agent-base for \`${req.method} ${req.path}\``
);
onerror(err);
};
if (typeof this.callback !== 'function') {
onerror(new Error('`callback` is not defined'));
return;
}
if (!this.promisifiedCallback) {
if (this.callback.length >= 3) {
debug('Converting legacy callback function to promise');
this.promisifiedCallback = promisify(this.callback);
} else {
this.promisifiedCallback = this.callback;
}
}
if (typeof timeoutMs === 'number' && timeoutMs > 0) {
timeoutId = setTimeout(ontimeout, timeoutMs);
}
if ('port' in opts && typeof opts.port !== 'number') {
opts.port = Number(opts.port);
}
try {
debug(
'Resolving socket for %o request: %o',
opts.protocol,
`${req.method} ${req.path}`
);
Promise.resolve(this.promisifiedCallback(req, opts)).then(
onsocket,
callbackError
);
} catch (err) {
Promise.reject(err).catch(callbackError);
}
}
freeSocket(socket: net.Socket, opts: AgentOptions) {
debug('Freeing socket %o %o', socket.constructor.name, opts);
socket.destroy();
}
destroy() {
debug('Destroying agent %o', this.constructor.name);
}
}
// So that `instanceof` works correctly
createAgent.prototype = createAgent.Agent.prototype;
}
export = createAgent;

View File

@@ -0,0 +1,33 @@
import {
Agent,
ClientRequest,
RequestOptions,
AgentCallbackCallback,
AgentCallbackPromise,
AgentCallbackReturn
} from './index';
type LegacyCallback = (
req: ClientRequest,
opts: RequestOptions,
fn: AgentCallbackCallback
) => void;
export default function promisify(fn: LegacyCallback): AgentCallbackPromise {
return function(this: Agent, req: ClientRequest, opts: RequestOptions) {
return new Promise((resolve, reject) => {
fn.call(
this,
req,
opts,
(err: Error | null | undefined, rtn?: AgentCallbackReturn) => {
if (err) {
reject(err);
} else {
resolve(rtn);
}
}
);
});
};
}

View File

@@ -0,0 +1,74 @@
http-proxy-agent
================
### An HTTP(s) proxy `http.Agent` implementation for HTTP
[![Build Status](https://github.com/TooTallNate/node-http-proxy-agent/workflows/Node%20CI/badge.svg)](https://github.com/TooTallNate/node-http-proxy-agent/actions?workflow=Node+CI)
This module provides an `http.Agent` implementation that connects to a specified
HTTP or HTTPS proxy server, and can be used with the built-in `http` module.
__Note:__ For HTTP proxy usage with the `https` module, check out
[`node-https-proxy-agent`](https://github.com/TooTallNate/node-https-proxy-agent).
Installation
------------
Install with `npm`:
``` bash
$ npm install http-proxy-agent
```
Example
-------
``` js
var url = require('url');
var http = require('http');
var HttpProxyAgent = require('http-proxy-agent');
// HTTP/HTTPS proxy to connect to
var proxy = process.env.http_proxy || 'http://168.63.76.32:3128';
console.log('using proxy server %j', proxy);
// HTTP endpoint for the proxy to connect to
var endpoint = process.argv[2] || 'http://nodejs.org/api/';
console.log('attempting to GET %j', endpoint);
var opts = url.parse(endpoint);
// create an instance of the `HttpProxyAgent` class with the proxy server information
var agent = new HttpProxyAgent(proxy);
opts.agent = agent;
http.get(opts, function (res) {
console.log('"response" event!', res.headers);
res.pipe(process.stdout);
});
```
License
-------
(The MIT License)
Copyright (c) 2013 Nathan Rajlich &lt;nathan@tootallnate.net&gt;
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@@ -0,0 +1,32 @@
/// <reference types="node" />
import net from 'net';
import { Agent, ClientRequest, RequestOptions } from 'agent-base';
import { HttpProxyAgentOptions } from '.';
interface HttpProxyAgentClientRequest extends ClientRequest {
path: string;
output?: string[];
outputData?: {
data: string;
}[];
_header?: string | null;
_implicitHeader(): void;
}
/**
* The `HttpProxyAgent` implements an HTTP Agent subclass that connects
* to the specified "HTTP proxy server" in order to proxy HTTP requests.
*
* @api public
*/
export default class HttpProxyAgent extends Agent {
private secureProxy;
private proxy;
constructor(_opts: string | HttpProxyAgentOptions);
/**
* Called when the node-core HTTP client library is creating a
* new HTTP request.
*
* @api protected
*/
callback(req: HttpProxyAgentClientRequest, opts: RequestOptions): Promise<net.Socket>;
}
export {};

View File

@@ -0,0 +1,145 @@
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const net_1 = __importDefault(require("net"));
const tls_1 = __importDefault(require("tls"));
const url_1 = __importDefault(require("url"));
const debug_1 = __importDefault(require("debug"));
const once_1 = __importDefault(require("@tootallnate/once"));
const agent_base_1 = require("agent-base");
const debug = (0, debug_1.default)('http-proxy-agent');
function isHTTPS(protocol) {
return typeof protocol === 'string' ? /^https:?$/i.test(protocol) : false;
}
/**
* The `HttpProxyAgent` implements an HTTP Agent subclass that connects
* to the specified "HTTP proxy server" in order to proxy HTTP requests.
*
* @api public
*/
class HttpProxyAgent extends agent_base_1.Agent {
constructor(_opts) {
let opts;
if (typeof _opts === 'string') {
opts = url_1.default.parse(_opts);
}
else {
opts = _opts;
}
if (!opts) {
throw new Error('an HTTP(S) proxy server `host` and `port` must be specified!');
}
debug('Creating new HttpProxyAgent instance: %o', opts);
super(opts);
const proxy = Object.assign({}, opts);
// If `true`, then connect to the proxy server over TLS.
// Defaults to `false`.
this.secureProxy = opts.secureProxy || isHTTPS(proxy.protocol);
// Prefer `hostname` over `host`, and set the `port` if needed.
proxy.host = proxy.hostname || proxy.host;
if (typeof proxy.port === 'string') {
proxy.port = parseInt(proxy.port, 10);
}
if (!proxy.port && proxy.host) {
proxy.port = this.secureProxy ? 443 : 80;
}
if (proxy.host && proxy.path) {
// If both a `host` and `path` are specified then it's most likely
// the result of a `url.parse()` call... we need to remove the
// `path` portion so that `net.connect()` doesn't attempt to open
// that as a Unix socket file.
delete proxy.path;
delete proxy.pathname;
}
this.proxy = proxy;
}
/**
* Called when the node-core HTTP client library is creating a
* new HTTP request.
*
* @api protected
*/
callback(req, opts) {
return __awaiter(this, void 0, void 0, function* () {
const { proxy, secureProxy } = this;
const parsed = url_1.default.parse(req.path);
if (!parsed.protocol) {
parsed.protocol = 'http:';
}
if (!parsed.hostname) {
parsed.hostname = opts.hostname || opts.host || null;
}
if (parsed.port == null && typeof opts.port) {
parsed.port = String(opts.port);
}
if (parsed.port === '80') {
// if port is 80, then we can remove the port so that the
// ":80" portion is not on the produced URL
parsed.port = '';
}
// Change the `http.ClientRequest` instance's "path" field
// to the absolute path of the URL that will be requested.
req.path = url_1.default.format(parsed);
// Inject the `Proxy-Authorization` header if necessary.
if (proxy.auth) {
req.setHeader('Proxy-Authorization', `Basic ${Buffer.from(proxy.auth).toString('base64')}`);
}
// Create a socket connection to the proxy server.
let socket;
if (secureProxy) {
debug('Creating `tls.Socket`: %o', proxy);
socket = tls_1.default.connect(proxy);
}
else {
debug('Creating `net.Socket`: %o', proxy);
socket = net_1.default.connect(proxy);
}
// At this point, the http ClientRequest's internal `_header` field
// might have already been set. If this is the case then we'll need
// to re-generate the string since we just changed the `req.path`.
if (req._header) {
let first;
let endOfHeaders;
debug('Regenerating stored HTTP header string for request');
req._header = null;
req._implicitHeader();
if (req.output && req.output.length > 0) {
// Node < 12
debug('Patching connection write() output buffer with updated header');
first = req.output[0];
endOfHeaders = first.indexOf('\r\n\r\n') + 4;
req.output[0] = req._header + first.substring(endOfHeaders);
debug('Output buffer: %o', req.output);
}
else if (req.outputData && req.outputData.length > 0) {
// Node >= 12
debug('Patching connection write() output buffer with updated header');
first = req.outputData[0].data;
endOfHeaders = first.indexOf('\r\n\r\n') + 4;
req.outputData[0].data =
req._header + first.substring(endOfHeaders);
debug('Output buffer: %o', req.outputData[0].data);
}
}
// Wait for the socket's `connect` event, so that this `callback()`
// function throws instead of the `http` request machinery. This is
// important for i.e. `PacProxyAgent` which determines a failed proxy
// connection via the `callback()` function throwing.
yield (0, once_1.default)(socket, 'connect');
return socket;
});
}
}
exports.default = HttpProxyAgent;
//# sourceMappingURL=agent.js.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"agent.js","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":";;;;;;;;;;;;;;AAAA,8CAAsB;AACtB,8CAAsB;AACtB,8CAAsB;AACtB,kDAAgC;AAChC,6DAAqC;AACrC,2CAAkE;AAGlE,MAAM,KAAK,GAAG,IAAA,eAAW,EAAC,kBAAkB,CAAC,CAAC;AAY9C,SAAS,OAAO,CAAC,QAAwB;IACxC,OAAO,OAAO,QAAQ,KAAK,QAAQ,CAAC,CAAC,CAAC,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC;AAC3E,CAAC;AAED;;;;;GAKG;AACH,MAAqB,cAAe,SAAQ,kBAAK;IAIhD,YAAY,KAAqC;QAChD,IAAI,IAA2B,CAAC;QAChC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE;YAC9B,IAAI,GAAG,aAAG,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;SACxB;aAAM;YACN,IAAI,GAAG,KAAK,CAAC;SACb;QACD,IAAI,CAAC,IAAI,EAAE;YACV,MAAM,IAAI,KAAK,CACd,8DAA8D,CAC9D,CAAC;SACF;QACD,KAAK,CAAC,0CAA0C,EAAE,IAAI,CAAC,CAAC;QACxD,KAAK,CAAC,IAAI,CAAC,CAAC;QAEZ,MAAM,KAAK,qBAA+B,IAAI,CAAE,CAAC;QAEjD,wDAAwD;QACxD,uBAAuB;QACvB,IAAI,CAAC,WAAW,GAAG,IAAI,CAAC,WAAW,IAAI,OAAO,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;QAE/D,+DAA+D;QAC/D,KAAK,CAAC,IAAI,GAAG,KAAK,CAAC,QAAQ,IAAI,KAAK,CAAC,IAAI,CAAC;QAC1C,IAAI,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,EAAE;YACnC,KAAK,CAAC,IAAI,GAAG,QAAQ,CAAC,KAAK,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;SACtC;QACD,IAAI,CAAC,KAAK,CAAC,IAAI,IAAI,KAAK,CAAC,IAAI,EAAE;YAC9B,KAAK,CAAC,IAAI,GAAG,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;SACzC;QAED,IAAI,KAAK,CAAC,IAAI,IAAI,KAAK,CAAC,IAAI,EAAE;YAC7B,kEAAkE;YAClE,8DAA8D;YAC9D,iEAAiE;YACjE,8BAA8B;YAC9B,OAAO,KAAK,CAAC,IAAI,CAAC;YAClB,OAAO,KAAK,CAAC,QAAQ,CAAC;SACtB;QAED,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACpB,CAAC;IAED;;;;;OAKG;IACG,QAAQ,CACb,GAAgC,EAChC,IAAoB;;YAEpB,MAAM,EAAE,KAAK,EAAE,WAAW,EAAE,GAAG,IAAI,CAAC;YACpC,MAAM,MAAM,GAAG,aAAG,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;YAEnC,IAAI,CAAC,MAAM,CAAC,QAAQ,EAAE;gBACrB,MAAM,CAAC,QAAQ,GAAG,OAAO,CAAC;aAC1B;YAED,IAAI,CAAC,MAAM,CAAC,QAAQ,EAAE;gBACrB,MAAM,CAAC,QAAQ,GAAG,IAAI,CAAC,QAAQ,IAAI,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC;aACrD;YAED,IAAI,MAAM,CAAC,IAAI,IAAI,IAAI,IAAI,OAAO,IAAI,CAAC,IAAI,EAAE;gBAC5C,MAAM,CAAC,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;aAChC;YAED,IAAI,MAAM,CAAC,IAAI,KAAK,IAAI,EAAE;gBACzB,yDAAyD;gBACzD,2CAA2C;gBAC3C,MAAM,CAAC,IAAI,GAAG,EAAE,CAAC;aACjB;YAED,0DAA0D;YAC1D,0DAA0D;YAC1D,GAAG,CAAC,IAAI,GAAG,aAAG,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC;YAE9B,wDAAwD;YACxD,IAAI,KAAK,CAAC,IAAI,EAAE;gBACf,GAAG,CAAC,SAAS,CACZ,qBAAqB,EACrB,SAAS,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE,CACrD,CAAC;aACF;YAED,kDAAkD;YAClD,IAAI,MAAkB,CAAC;YACvB,IAAI,WAAW,EAAE;gBAChB,KAAK,CAAC,2BAA2B,EAAE,KAAK,CAAC,CAAC;gBAC1C,MAAM,GAAG,aAAG,CAAC,OAAO,CAAC,KAA8B,CAAC,CAAC;aACrD;iBAAM;gBACN,KAAK,CAAC,2BAA2B,EAAE,KAAK,CAAC,CAAC;gBAC1C,MAAM,GAAG,aAAG,CAAC,OAAO,CAAC,KAA2B,CAAC,CAAC;aAClD;YAED,mEAAmE;YACnE,mEAAmE;YACnE,kEAAkE;YAClE,IAAI,GAAG,CAAC,OAAO,EAAE;gBAChB,IAAI,KAAa,CAAC;gBAClB,IAAI,YAAoB,CAAC;gBACzB,KAAK,CAAC,oDAAoD,CAAC,CAAC;gBAC5D,GAAG,CAAC,OAAO,GAAG,IAAI,CAAC;gBACnB,GAAG,CAAC,eAAe,EAAE,CAAC;gBACtB,IAAI,GAAG,CAAC,MAAM,IAAI,GAAG,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,EAAE;oBACxC,YAAY;oBACZ,KAAK,CACJ,+DAA+D,CAC/D,CAAC;oBACF,KAAK,GAAG,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC;oBACtB,YAAY,GAAG,KAAK,CAAC,OAAO,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC;oBAC7C,GAAG,CAAC,MAAM,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,OAAO,GAAG,KAAK,CAAC,SAAS,CAAC,YAAY,CAAC,CAAC;oBAC5D,KAAK,CAAC,mBAAmB,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;iBACvC;qBAAM,IAAI,GAAG,CAAC,UAAU,IAAI,GAAG,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC,EAAE;oBACvD,aAAa;oBACb,KAAK,CACJ,+DAA+D,CAC/D,CAAC;oBACF,KAAK,GAAG,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC;oBAC/B,YAAY,GAAG,KAAK,CAAC,OAAO,CAAC,UAAU,CAAC,GAAG,CAAC,CAAC;oBAC7C,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,IAAI;wBACrB,GAAG,CAAC,OAAO,GAAG,KAAK,CAAC,SAAS,CAAC,YAAY,CAAC,CAAC;oBAC7C,KAAK,CAAC,mBAAmB,EAAE,GAAG,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;iBACnD;aACD;YAED,mEAAmE;YACnE,mEAAmE;YACnE,qEAAqE;YACrE,qDAAqD;YACrD,MAAM,IAAA,cAAI,EAAC,MAAM,EAAE,SAAS,CAAC,CAAC;YAE9B,OAAO,MAAM,CAAC;QACf,CAAC;KAAA;CACD;AA1ID,iCA0IC"}

View File

@@ -0,0 +1,21 @@
/// <reference types="node" />
import net from 'net';
import tls from 'tls';
import { Url } from 'url';
import { AgentOptions } from 'agent-base';
import _HttpProxyAgent from './agent';
declare function createHttpProxyAgent(opts: string | createHttpProxyAgent.HttpProxyAgentOptions): _HttpProxyAgent;
declare namespace createHttpProxyAgent {
interface BaseHttpProxyAgentOptions {
secureProxy?: boolean;
host?: string | null;
path?: string | null;
port?: string | number | null;
}
export interface HttpProxyAgentOptions extends AgentOptions, BaseHttpProxyAgentOptions, Partial<Omit<Url & net.NetConnectOpts & tls.ConnectionOptions, keyof BaseHttpProxyAgentOptions>> {
}
export type HttpProxyAgent = _HttpProxyAgent;
export const HttpProxyAgent: typeof _HttpProxyAgent;
export {};
}
export = createHttpProxyAgent;

View File

@@ -0,0 +1,14 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
const agent_1 = __importDefault(require("./agent"));
function createHttpProxyAgent(opts) {
return new agent_1.default(opts);
}
(function (createHttpProxyAgent) {
createHttpProxyAgent.HttpProxyAgent = agent_1.default;
createHttpProxyAgent.prototype = agent_1.default.prototype;
})(createHttpProxyAgent || (createHttpProxyAgent = {}));
module.exports = createHttpProxyAgent;
//# sourceMappingURL=index.js.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";;;;AAIA,oDAAsC;AAEtC,SAAS,oBAAoB,CAC5B,IAAyD;IAEzD,OAAO,IAAI,eAAe,CAAC,IAAI,CAAC,CAAC;AAClC,CAAC;AAED,WAAU,oBAAoB;IAmBhB,mCAAc,GAAG,eAAe,CAAC;IAE9C,oBAAoB,CAAC,SAAS,GAAG,eAAe,CAAC,SAAS,CAAC;AAC5D,CAAC,EAtBS,oBAAoB,KAApB,oBAAoB,QAsB7B;AAED,iBAAS,oBAAoB,CAAC"}

View File

@@ -0,0 +1,57 @@
{
"name": "http-proxy-agent",
"version": "5.0.0",
"description": "An HTTP(s) proxy `http.Agent` implementation for HTTP",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"files": [
"dist"
],
"scripts": {
"prebuild": "rimraf dist",
"build": "tsc",
"test": "mocha",
"test-lint": "eslint src --ext .js,.ts",
"prepublishOnly": "npm run build"
},
"repository": {
"type": "git",
"url": "git://github.com/TooTallNate/node-http-proxy-agent.git"
},
"keywords": [
"http",
"proxy",
"endpoint",
"agent"
],
"author": "Nathan Rajlich <nathan@tootallnate.net> (http://n8.io/)",
"license": "MIT",
"bugs": {
"url": "https://github.com/TooTallNate/node-http-proxy-agent/issues"
},
"dependencies": {
"@tootallnate/once": "2",
"agent-base": "6",
"debug": "4"
},
"devDependencies": {
"@types/debug": "4",
"@types/node": "^12.19.2",
"@typescript-eslint/eslint-plugin": "1.6.0",
"@typescript-eslint/parser": "1.1.0",
"eslint": "5.16.0",
"eslint-config-airbnb": "17.1.0",
"eslint-config-prettier": "4.1.0",
"eslint-import-resolver-typescript": "1.1.1",
"eslint-plugin-import": "2.16.0",
"eslint-plugin-jsx-a11y": "6.2.1",
"eslint-plugin-react": "7.12.4",
"mocha": "^6.2.2",
"proxy": "1",
"rimraf": "^3.0.0",
"typescript": "^4.4.3"
},
"engines": {
"node": ">= 6"
}
}

View File

@@ -0,0 +1,137 @@
https-proxy-agent
================
### An HTTP(s) proxy `http.Agent` implementation for HTTPS
[![Build Status](https://github.com/TooTallNate/node-https-proxy-agent/workflows/Node%20CI/badge.svg)](https://github.com/TooTallNate/node-https-proxy-agent/actions?workflow=Node+CI)
This module provides an `http.Agent` implementation that connects to a specified
HTTP or HTTPS proxy server, and can be used with the built-in `https` module.
Specifically, this `Agent` implementation connects to an intermediary "proxy"
server and issues the [CONNECT HTTP method][CONNECT], which tells the proxy to
open a direct TCP connection to the destination server.
Since this agent implements the CONNECT HTTP method, it also works with other
protocols that use this method when connecting over proxies (i.e. WebSockets).
See the "Examples" section below for more.
Installation
------------
Install with `npm`:
``` bash
$ npm install https-proxy-agent
```
Examples
--------
#### `https` module example
``` js
var url = require('url');
var https = require('https');
var HttpsProxyAgent = require('https-proxy-agent');
// HTTP/HTTPS proxy to connect to
var proxy = process.env.http_proxy || 'http://168.63.76.32:3128';
console.log('using proxy server %j', proxy);
// HTTPS endpoint for the proxy to connect to
var endpoint = process.argv[2] || 'https://graph.facebook.com/tootallnate';
console.log('attempting to GET %j', endpoint);
var options = url.parse(endpoint);
// create an instance of the `HttpsProxyAgent` class with the proxy server information
var agent = new HttpsProxyAgent(proxy);
options.agent = agent;
https.get(options, function (res) {
console.log('"response" event!', res.headers);
res.pipe(process.stdout);
});
```
#### `ws` WebSocket connection example
``` js
var url = require('url');
var WebSocket = require('ws');
var HttpsProxyAgent = require('https-proxy-agent');
// HTTP/HTTPS proxy to connect to
var proxy = process.env.http_proxy || 'http://168.63.76.32:3128';
console.log('using proxy server %j', proxy);
// WebSocket endpoint for the proxy to connect to
var endpoint = process.argv[2] || 'ws://echo.websocket.org';
var parsed = url.parse(endpoint);
console.log('attempting to connect to WebSocket %j', endpoint);
// create an instance of the `HttpsProxyAgent` class with the proxy server information
var options = url.parse(proxy);
var agent = new HttpsProxyAgent(options);
// finally, initiate the WebSocket connection
var socket = new WebSocket(endpoint, { agent: agent });
socket.on('open', function () {
console.log('"open" event!');
socket.send('hello world');
});
socket.on('message', function (data, flags) {
console.log('"message" event! %j %j', data, flags);
socket.close();
});
```
API
---
### new HttpsProxyAgent(Object options)
The `HttpsProxyAgent` class implements an `http.Agent` subclass that connects
to the specified "HTTP(s) proxy server" in order to proxy HTTPS and/or WebSocket
requests. This is achieved by using the [HTTP `CONNECT` method][CONNECT].
The `options` argument may either be a string URI of the proxy server to use, or an
"options" object with more specific properties:
* `host` - String - Proxy host to connect to (may use `hostname` as well). Required.
* `port` - Number - Proxy port to connect to. Required.
* `protocol` - String - If `https:`, then use TLS to connect to the proxy.
* `headers` - Object - Additional HTTP headers to be sent on the HTTP CONNECT method.
* Any other options given are passed to the `net.connect()`/`tls.connect()` functions.
License
-------
(The MIT License)
Copyright (c) 2013 Nathan Rajlich &lt;nathan@tootallnate.net&gt;
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
[CONNECT]: http://en.wikipedia.org/wiki/HTTP_tunnel#HTTP_CONNECT_Tunneling

View File

@@ -0,0 +1,30 @@
/// <reference types="node" />
import net from 'net';
import { Agent, ClientRequest, RequestOptions } from 'agent-base';
import { HttpsProxyAgentOptions } from '.';
/**
* The `HttpsProxyAgent` implements an HTTP Agent subclass that connects to
* the specified "HTTP(s) proxy server" in order to proxy HTTPS requests.
*
* Outgoing HTTP requests are first tunneled through the proxy server using the
* `CONNECT` HTTP request method to establish a connection to the proxy server,
* and then the proxy server connects to the destination target and issues the
* HTTP request from the proxy server.
*
* `https:` requests have their socket connection upgraded to TLS once
* the connection to the proxy server has been established.
*
* @api public
*/
export default class HttpsProxyAgent extends Agent {
private secureProxy;
private proxy;
constructor(_opts: string | HttpsProxyAgentOptions);
/**
* Called when the node-core HTTP client library is creating a
* new HTTP request.
*
* @api protected
*/
callback(req: ClientRequest, opts: RequestOptions): Promise<net.Socket>;
}

View File

@@ -0,0 +1,177 @@
"use strict";
var __awaiter = (this && this.__awaiter) || function (thisArg, _arguments, P, generator) {
function adopt(value) { return value instanceof P ? value : new P(function (resolve) { resolve(value); }); }
return new (P || (P = Promise))(function (resolve, reject) {
function fulfilled(value) { try { step(generator.next(value)); } catch (e) { reject(e); } }
function rejected(value) { try { step(generator["throw"](value)); } catch (e) { reject(e); } }
function step(result) { result.done ? resolve(result.value) : adopt(result.value).then(fulfilled, rejected); }
step((generator = generator.apply(thisArg, _arguments || [])).next());
});
};
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const net_1 = __importDefault(require("net"));
const tls_1 = __importDefault(require("tls"));
const url_1 = __importDefault(require("url"));
const assert_1 = __importDefault(require("assert"));
const debug_1 = __importDefault(require("debug"));
const agent_base_1 = require("agent-base");
const parse_proxy_response_1 = __importDefault(require("./parse-proxy-response"));
const debug = debug_1.default('https-proxy-agent:agent');
/**
* The `HttpsProxyAgent` implements an HTTP Agent subclass that connects to
* the specified "HTTP(s) proxy server" in order to proxy HTTPS requests.
*
* Outgoing HTTP requests are first tunneled through the proxy server using the
* `CONNECT` HTTP request method to establish a connection to the proxy server,
* and then the proxy server connects to the destination target and issues the
* HTTP request from the proxy server.
*
* `https:` requests have their socket connection upgraded to TLS once
* the connection to the proxy server has been established.
*
* @api public
*/
class HttpsProxyAgent extends agent_base_1.Agent {
constructor(_opts) {
let opts;
if (typeof _opts === 'string') {
opts = url_1.default.parse(_opts);
}
else {
opts = _opts;
}
if (!opts) {
throw new Error('an HTTP(S) proxy server `host` and `port` must be specified!');
}
debug('creating new HttpsProxyAgent instance: %o', opts);
super(opts);
const proxy = Object.assign({}, opts);
// If `true`, then connect to the proxy server over TLS.
// Defaults to `false`.
this.secureProxy = opts.secureProxy || isHTTPS(proxy.protocol);
// Prefer `hostname` over `host`, and set the `port` if needed.
proxy.host = proxy.hostname || proxy.host;
if (typeof proxy.port === 'string') {
proxy.port = parseInt(proxy.port, 10);
}
if (!proxy.port && proxy.host) {
proxy.port = this.secureProxy ? 443 : 80;
}
// ALPN is supported by Node.js >= v5.
// attempt to negotiate http/1.1 for proxy servers that support http/2
if (this.secureProxy && !('ALPNProtocols' in proxy)) {
proxy.ALPNProtocols = ['http 1.1'];
}
if (proxy.host && proxy.path) {
// If both a `host` and `path` are specified then it's most likely
// the result of a `url.parse()` call... we need to remove the
// `path` portion so that `net.connect()` doesn't attempt to open
// that as a Unix socket file.
delete proxy.path;
delete proxy.pathname;
}
this.proxy = proxy;
}
/**
* Called when the node-core HTTP client library is creating a
* new HTTP request.
*
* @api protected
*/
callback(req, opts) {
return __awaiter(this, void 0, void 0, function* () {
const { proxy, secureProxy } = this;
// Create a socket connection to the proxy server.
let socket;
if (secureProxy) {
debug('Creating `tls.Socket`: %o', proxy);
socket = tls_1.default.connect(proxy);
}
else {
debug('Creating `net.Socket`: %o', proxy);
socket = net_1.default.connect(proxy);
}
const headers = Object.assign({}, proxy.headers);
const hostname = `${opts.host}:${opts.port}`;
let payload = `CONNECT ${hostname} HTTP/1.1\r\n`;
// Inject the `Proxy-Authorization` header if necessary.
if (proxy.auth) {
headers['Proxy-Authorization'] = `Basic ${Buffer.from(proxy.auth).toString('base64')}`;
}
// The `Host` header should only include the port
// number when it is not the default port.
let { host, port, secureEndpoint } = opts;
if (!isDefaultPort(port, secureEndpoint)) {
host += `:${port}`;
}
headers.Host = host;
headers.Connection = 'close';
for (const name of Object.keys(headers)) {
payload += `${name}: ${headers[name]}\r\n`;
}
const proxyResponsePromise = parse_proxy_response_1.default(socket);
socket.write(`${payload}\r\n`);
const { statusCode, buffered } = yield proxyResponsePromise;
if (statusCode === 200) {
req.once('socket', resume);
if (opts.secureEndpoint) {
// The proxy is connecting to a TLS server, so upgrade
// this socket connection to a TLS connection.
debug('Upgrading socket connection to TLS');
const servername = opts.servername || opts.host;
return tls_1.default.connect(Object.assign(Object.assign({}, omit(opts, 'host', 'hostname', 'path', 'port')), { socket,
servername }));
}
return socket;
}
// Some other status code that's not 200... need to re-play the HTTP
// header "data" events onto the socket once the HTTP machinery is
// attached so that the node core `http` can parse and handle the
// error status code.
// Close the original socket, and a new "fake" socket is returned
// instead, so that the proxy doesn't get the HTTP request
// written to it (which may contain `Authorization` headers or other
// sensitive data).
//
// See: https://hackerone.com/reports/541502
socket.destroy();
const fakeSocket = new net_1.default.Socket({ writable: false });
fakeSocket.readable = true;
// Need to wait for the "socket" event to re-play the "data" events.
req.once('socket', (s) => {
debug('replaying proxy buffer for failed request');
assert_1.default(s.listenerCount('data') > 0);
// Replay the "buffered" Buffer onto the fake `socket`, since at
// this point the HTTP module machinery has been hooked up for
// the user.
s.push(buffered);
s.push(null);
});
return fakeSocket;
});
}
}
exports.default = HttpsProxyAgent;
function resume(socket) {
socket.resume();
}
function isDefaultPort(port, secure) {
return Boolean((!secure && port === 80) || (secure && port === 443));
}
function isHTTPS(protocol) {
return typeof protocol === 'string' ? /^https:?$/i.test(protocol) : false;
}
function omit(obj, ...keys) {
const ret = {};
let key;
for (key in obj) {
if (!keys.includes(key)) {
ret[key] = obj[key];
}
}
return ret;
}
//# sourceMappingURL=agent.js.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"agent.js","sourceRoot":"","sources":["../src/agent.ts"],"names":[],"mappings":";;;;;;;;;;;;;;AAAA,8CAAsB;AACtB,8CAAsB;AACtB,8CAAsB;AACtB,oDAA4B;AAC5B,kDAAgC;AAEhC,2CAAkE;AAElE,kFAAwD;AAExD,MAAM,KAAK,GAAG,eAAW,CAAC,yBAAyB,CAAC,CAAC;AAErD;;;;;;;;;;;;;GAaG;AACH,MAAqB,eAAgB,SAAQ,kBAAK;IAIjD,YAAY,KAAsC;QACjD,IAAI,IAA4B,CAAC;QACjC,IAAI,OAAO,KAAK,KAAK,QAAQ,EAAE;YAC9B,IAAI,GAAG,aAAG,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;SACxB;aAAM;YACN,IAAI,GAAG,KAAK,CAAC;SACb;QACD,IAAI,CAAC,IAAI,EAAE;YACV,MAAM,IAAI,KAAK,CACd,8DAA8D,CAC9D,CAAC;SACF;QACD,KAAK,CAAC,2CAA2C,EAAE,IAAI,CAAC,CAAC;QACzD,KAAK,CAAC,IAAI,CAAC,CAAC;QAEZ,MAAM,KAAK,qBAAgC,IAAI,CAAE,CAAC;QAElD,wDAAwD;QACxD,uBAAuB;QACvB,IAAI,CAAC,WAAW,GAAG,IAAI,CAAC,WAAW,IAAI,OAAO,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;QAE/D,+DAA+D;QAC/D,KAAK,CAAC,IAAI,GAAG,KAAK,CAAC,QAAQ,IAAI,KAAK,CAAC,IAAI,CAAC;QAC1C,IAAI,OAAO,KAAK,CAAC,IAAI,KAAK,QAAQ,EAAE;YACnC,KAAK,CAAC,IAAI,GAAG,QAAQ,CAAC,KAAK,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;SACtC;QACD,IAAI,CAAC,KAAK,CAAC,IAAI,IAAI,KAAK,CAAC,IAAI,EAAE;YAC9B,KAAK,CAAC,IAAI,GAAG,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC;SACzC;QAED,sCAAsC;QACtC,sEAAsE;QACtE,IAAI,IAAI,CAAC,WAAW,IAAI,CAAC,CAAC,eAAe,IAAI,KAAK,CAAC,EAAE;YACpD,KAAK,CAAC,aAAa,GAAG,CAAC,UAAU,CAAC,CAAC;SACnC;QAED,IAAI,KAAK,CAAC,IAAI,IAAI,KAAK,CAAC,IAAI,EAAE;YAC7B,kEAAkE;YAClE,8DAA8D;YAC9D,iEAAiE;YACjE,8BAA8B;YAC9B,OAAO,KAAK,CAAC,IAAI,CAAC;YAClB,OAAO,KAAK,CAAC,QAAQ,CAAC;SACtB;QAED,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACpB,CAAC;IAED;;;;;OAKG;IACG,QAAQ,CACb,GAAkB,EAClB,IAAoB;;YAEpB,MAAM,EAAE,KAAK,EAAE,WAAW,EAAE,GAAG,IAAI,CAAC;YAEpC,kDAAkD;YAClD,IAAI,MAAkB,CAAC;YACvB,IAAI,WAAW,EAAE;gBAChB,KAAK,CAAC,2BAA2B,EAAE,KAAK,CAAC,CAAC;gBAC1C,MAAM,GAAG,aAAG,CAAC,OAAO,CAAC,KAA8B,CAAC,CAAC;aACrD;iBAAM;gBACN,KAAK,CAAC,2BAA2B,EAAE,KAAK,CAAC,CAAC;gBAC1C,MAAM,GAAG,aAAG,CAAC,OAAO,CAAC,KAA2B,CAAC,CAAC;aAClD;YAED,MAAM,OAAO,qBAA6B,KAAK,CAAC,OAAO,CAAE,CAAC;YAC1D,MAAM,QAAQ,GAAG,GAAG,IAAI,CAAC,IAAI,IAAI,IAAI,CAAC,IAAI,EAAE,CAAC;YAC7C,IAAI,OAAO,GAAG,WAAW,QAAQ,eAAe,CAAC;YAEjD,wDAAwD;YACxD,IAAI,KAAK,CAAC,IAAI,EAAE;gBACf,OAAO,CAAC,qBAAqB,CAAC,GAAG,SAAS,MAAM,CAAC,IAAI,CACpD,KAAK,CAAC,IAAI,CACV,CAAC,QAAQ,CAAC,QAAQ,CAAC,EAAE,CAAC;aACvB;YAED,iDAAiD;YACjD,0CAA0C;YAC1C,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,cAAc,EAAE,GAAG,IAAI,CAAC;YAC1C,IAAI,CAAC,aAAa,CAAC,IAAI,EAAE,cAAc,CAAC,EAAE;gBACzC,IAAI,IAAI,IAAI,IAAI,EAAE,CAAC;aACnB;YACD,OAAO,CAAC,IAAI,GAAG,IAAI,CAAC;YAEpB,OAAO,CAAC,UAAU,GAAG,OAAO,CAAC;YAC7B,KAAK,MAAM,IAAI,IAAI,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE;gBACxC,OAAO,IAAI,GAAG,IAAI,KAAK,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC;aAC3C;YAED,MAAM,oBAAoB,GAAG,8BAAkB,CAAC,MAAM,CAAC,CAAC;YAExD,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,MAAM,CAAC,CAAC;YAE/B,MAAM,EACL,UAAU,EACV,QAAQ,EACR,GAAG,MAAM,oBAAoB,CAAC;YAE/B,IAAI,UAAU,KAAK,GAAG,EAAE;gBACvB,GAAG,CAAC,IAAI,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAC;gBAE3B,IAAI,IAAI,CAAC,cAAc,EAAE;oBACxB,sDAAsD;oBACtD,8CAA8C;oBAC9C,KAAK,CAAC,oCAAoC,CAAC,CAAC;oBAC5C,MAAM,UAAU,GAAG,IAAI,CAAC,UAAU,IAAI,IAAI,CAAC,IAAI,CAAC;oBAChD,OAAO,aAAG,CAAC,OAAO,iCACd,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,EAAE,MAAM,CAAC,KACjD,MAAM;wBACN,UAAU,IACT,CAAC;iBACH;gBAED,OAAO,MAAM,CAAC;aACd;YAED,oEAAoE;YACpE,kEAAkE;YAClE,iEAAiE;YACjE,qBAAqB;YAErB,iEAAiE;YACjE,0DAA0D;YAC1D,oEAAoE;YACpE,mBAAmB;YACnB,EAAE;YACF,4CAA4C;YAC5C,MAAM,CAAC,OAAO,EAAE,CAAC;YAEjB,MAAM,UAAU,GAAG,IAAI,aAAG,CAAC,MAAM,CAAC,EAAE,QAAQ,EAAE,KAAK,EAAE,CAAC,CAAC;YACvD,UAAU,CAAC,QAAQ,GAAG,IAAI,CAAC;YAE3B,oEAAoE;YACpE,GAAG,CAAC,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAa,EAAE,EAAE;gBACpC,KAAK,CAAC,2CAA2C,CAAC,CAAC;gBACnD,gBAAM,CAAC,CAAC,CAAC,aAAa,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC;gBAEpC,gEAAgE;gBAChE,8DAA8D;gBAC9D,YAAY;gBACZ,CAAC,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;gBACjB,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACd,CAAC,CAAC,CAAC;YAEH,OAAO,UAAU,CAAC;QACnB,CAAC;KAAA;CACD;AA3JD,kCA2JC;AAED,SAAS,MAAM,CAAC,MAAkC;IACjD,MAAM,CAAC,MAAM,EAAE,CAAC;AACjB,CAAC;AAED,SAAS,aAAa,CAAC,IAAY,EAAE,MAAe;IACnD,OAAO,OAAO,CAAC,CAAC,CAAC,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC,IAAI,CAAC,MAAM,IAAI,IAAI,KAAK,GAAG,CAAC,CAAC,CAAC;AACtE,CAAC;AAED,SAAS,OAAO,CAAC,QAAwB;IACxC,OAAO,OAAO,QAAQ,KAAK,QAAQ,CAAC,CAAC,CAAC,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC;AAC3E,CAAC;AAED,SAAS,IAAI,CACZ,GAAM,EACN,GAAG,IAAO;IAIV,MAAM,GAAG,GAAG,EAEX,CAAC;IACF,IAAI,GAAqB,CAAC;IAC1B,KAAK,GAAG,IAAI,GAAG,EAAE;QAChB,IAAI,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE;YACxB,GAAG,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC,GAAG,CAAC,CAAC;SACpB;KACD;IACD,OAAO,GAAG,CAAC;AACZ,CAAC"}

View File

@@ -0,0 +1,23 @@
/// <reference types="node" />
import net from 'net';
import tls from 'tls';
import { Url } from 'url';
import { AgentOptions } from 'agent-base';
import { OutgoingHttpHeaders } from 'http';
import _HttpsProxyAgent from './agent';
declare function createHttpsProxyAgent(opts: string | createHttpsProxyAgent.HttpsProxyAgentOptions): _HttpsProxyAgent;
declare namespace createHttpsProxyAgent {
interface BaseHttpsProxyAgentOptions {
headers?: OutgoingHttpHeaders;
secureProxy?: boolean;
host?: string | null;
path?: string | null;
port?: string | number | null;
}
export interface HttpsProxyAgentOptions extends AgentOptions, BaseHttpsProxyAgentOptions, Partial<Omit<Url & net.NetConnectOpts & tls.ConnectionOptions, keyof BaseHttpsProxyAgentOptions>> {
}
export type HttpsProxyAgent = _HttpsProxyAgent;
export const HttpsProxyAgent: typeof _HttpsProxyAgent;
export {};
}
export = createHttpsProxyAgent;

View File

@@ -0,0 +1,14 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
const agent_1 = __importDefault(require("./agent"));
function createHttpsProxyAgent(opts) {
return new agent_1.default(opts);
}
(function (createHttpsProxyAgent) {
createHttpsProxyAgent.HttpsProxyAgent = agent_1.default;
createHttpsProxyAgent.prototype = agent_1.default.prototype;
})(createHttpsProxyAgent || (createHttpsProxyAgent = {}));
module.exports = createHttpsProxyAgent;
//# sourceMappingURL=index.js.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";;;;AAKA,oDAAuC;AAEvC,SAAS,qBAAqB,CAC7B,IAA2D;IAE3D,OAAO,IAAI,eAAgB,CAAC,IAAI,CAAC,CAAC;AACnC,CAAC;AAED,WAAU,qBAAqB;IAoBjB,qCAAe,GAAG,eAAgB,CAAC;IAEhD,qBAAqB,CAAC,SAAS,GAAG,eAAgB,CAAC,SAAS,CAAC;AAC9D,CAAC,EAvBS,qBAAqB,KAArB,qBAAqB,QAuB9B;AAED,iBAAS,qBAAqB,CAAC"}

View File

@@ -0,0 +1,7 @@
/// <reference types="node" />
import { Readable } from 'stream';
export interface ProxyResponse {
statusCode: number;
buffered: Buffer;
}
export default function parseProxyResponse(socket: Readable): Promise<ProxyResponse>;

View File

@@ -0,0 +1,66 @@
"use strict";
var __importDefault = (this && this.__importDefault) || function (mod) {
return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", { value: true });
const debug_1 = __importDefault(require("debug"));
const debug = debug_1.default('https-proxy-agent:parse-proxy-response');
function parseProxyResponse(socket) {
return new Promise((resolve, reject) => {
// we need to buffer any HTTP traffic that happens with the proxy before we get
// the CONNECT response, so that if the response is anything other than an "200"
// response code, then we can re-play the "data" events on the socket once the
// HTTP parser is hooked up...
let buffersLength = 0;
const buffers = [];
function read() {
const b = socket.read();
if (b)
ondata(b);
else
socket.once('readable', read);
}
function cleanup() {
socket.removeListener('end', onend);
socket.removeListener('error', onerror);
socket.removeListener('close', onclose);
socket.removeListener('readable', read);
}
function onclose(err) {
debug('onclose had error %o', err);
}
function onend() {
debug('onend');
}
function onerror(err) {
cleanup();
debug('onerror %o', err);
reject(err);
}
function ondata(b) {
buffers.push(b);
buffersLength += b.length;
const buffered = Buffer.concat(buffers, buffersLength);
const endOfHeaders = buffered.indexOf('\r\n\r\n');
if (endOfHeaders === -1) {
// keep buffering
debug('have not received end of HTTP headers yet...');
read();
return;
}
const firstLine = buffered.toString('ascii', 0, buffered.indexOf('\r\n'));
const statusCode = +firstLine.split(' ')[1];
debug('got proxy server response: %o', firstLine);
resolve({
statusCode,
buffered
});
}
socket.on('error', onerror);
socket.on('close', onclose);
socket.on('end', onend);
read();
});
}
exports.default = parseProxyResponse;
//# sourceMappingURL=parse-proxy-response.js.map

View File

@@ -0,0 +1 @@
{"version":3,"file":"parse-proxy-response.js","sourceRoot":"","sources":["../src/parse-proxy-response.ts"],"names":[],"mappings":";;;;;AAAA,kDAAgC;AAGhC,MAAM,KAAK,GAAG,eAAW,CAAC,wCAAwC,CAAC,CAAC;AAOpE,SAAwB,kBAAkB,CACzC,MAAgB;IAEhB,OAAO,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;QACtC,+EAA+E;QAC/E,gFAAgF;QAChF,8EAA8E;QAC9E,8BAA8B;QAC9B,IAAI,aAAa,GAAG,CAAC,CAAC;QACtB,MAAM,OAAO,GAAa,EAAE,CAAC;QAE7B,SAAS,IAAI;YACZ,MAAM,CAAC,GAAG,MAAM,CAAC,IAAI,EAAE,CAAC;YACxB,IAAI,CAAC;gBAAE,MAAM,CAAC,CAAC,CAAC,CAAC;;gBACZ,MAAM,CAAC,IAAI,CAAC,UAAU,EAAE,IAAI,CAAC,CAAC;QACpC,CAAC;QAED,SAAS,OAAO;YACf,MAAM,CAAC,cAAc,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;YACpC,MAAM,CAAC,cAAc,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;YACxC,MAAM,CAAC,cAAc,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;YACxC,MAAM,CAAC,cAAc,CAAC,UAAU,EAAE,IAAI,CAAC,CAAC;QACzC,CAAC;QAED,SAAS,OAAO,CAAC,GAAW;YAC3B,KAAK,CAAC,sBAAsB,EAAE,GAAG,CAAC,CAAC;QACpC,CAAC;QAED,SAAS,KAAK;YACb,KAAK,CAAC,OAAO,CAAC,CAAC;QAChB,CAAC;QAED,SAAS,OAAO,CAAC,GAAU;YAC1B,OAAO,EAAE,CAAC;YACV,KAAK,CAAC,YAAY,EAAE,GAAG,CAAC,CAAC;YACzB,MAAM,CAAC,GAAG,CAAC,CAAC;QACb,CAAC;QAED,SAAS,MAAM,CAAC,CAAS;YACxB,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;YAChB,aAAa,IAAI,CAAC,CAAC,MAAM,CAAC;YAE1B,MAAM,QAAQ,GAAG,MAAM,CAAC,MAAM,CAAC,OAAO,EAAE,aAAa,CAAC,CAAC;YACvD,MAAM,YAAY,GAAG,QAAQ,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;YAElD,IAAI,YAAY,KAAK,CAAC,CAAC,EAAE;gBACxB,iBAAiB;gBACjB,KAAK,CAAC,8CAA8C,CAAC,CAAC;gBACtD,IAAI,EAAE,CAAC;gBACP,OAAO;aACP;YAED,MAAM,SAAS,GAAG,QAAQ,CAAC,QAAQ,CAClC,OAAO,EACP,CAAC,EACD,QAAQ,CAAC,OAAO,CAAC,MAAM,CAAC,CACxB,CAAC;YACF,MAAM,UAAU,GAAG,CAAC,SAAS,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC;YAC5C,KAAK,CAAC,+BAA+B,EAAE,SAAS,CAAC,CAAC;YAClD,OAAO,CAAC;gBACP,UAAU;gBACV,QAAQ;aACR,CAAC,CAAC;QACJ,CAAC;QAED,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QAC5B,MAAM,CAAC,EAAE,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QAC5B,MAAM,CAAC,EAAE,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;QAExB,IAAI,EAAE,CAAC;IACR,CAAC,CAAC,CAAC;AACJ,CAAC;AAvED,qCAuEC"}

View File

@@ -0,0 +1,56 @@
{
"name": "https-proxy-agent",
"version": "5.0.1",
"description": "An HTTP(s) proxy `http.Agent` implementation for HTTPS",
"main": "dist/index",
"types": "dist/index",
"files": [
"dist"
],
"scripts": {
"prebuild": "rimraf dist",
"build": "tsc",
"test": "mocha --reporter spec",
"test-lint": "eslint src --ext .js,.ts",
"prepublishOnly": "npm run build"
},
"repository": {
"type": "git",
"url": "git://github.com/TooTallNate/node-https-proxy-agent.git"
},
"keywords": [
"https",
"proxy",
"endpoint",
"agent"
],
"author": "Nathan Rajlich <nathan@tootallnate.net> (http://n8.io/)",
"license": "MIT",
"bugs": {
"url": "https://github.com/TooTallNate/node-https-proxy-agent/issues"
},
"dependencies": {
"agent-base": "6",
"debug": "4"
},
"devDependencies": {
"@types/debug": "4",
"@types/node": "^12.12.11",
"@typescript-eslint/eslint-plugin": "1.6.0",
"@typescript-eslint/parser": "1.1.0",
"eslint": "5.16.0",
"eslint-config-airbnb": "17.1.0",
"eslint-config-prettier": "4.1.0",
"eslint-import-resolver-typescript": "1.1.1",
"eslint-plugin-import": "2.16.0",
"eslint-plugin-jsx-a11y": "6.2.1",
"eslint-plugin-react": "7.12.4",
"mocha": "^6.2.2",
"proxy": "1",
"rimraf": "^3.0.0",
"typescript": "^3.5.3"
},
"engines": {
"node": ">= 6"
}
}

View File

@@ -0,0 +1,15 @@
The ISC License
Copyright (c) 2010-2023 Isaac Z. Schlueter and Contributors
Permission to use, copy, modify, and/or distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR
IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,869 @@
// Project: https://github.com/isaacs/node-lru-cache
// Based initially on @types/lru-cache
// https://github.com/DefinitelyTyped/DefinitelyTyped
// used under the terms of the MIT License, shown below.
//
// DefinitelyTyped license:
// ------
// MIT License
//
// Copyright (c) Microsoft Corporation.
//
// Permission is hereby granted, free of charge, to any person obtaining a
// copy of this software and associated documentation files (the "Software"),
// to deal in the Software without restriction, including without limitation
// the rights to use, copy, modify, merge, publish, distribute, sublicense,
// and/or sell copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included
// in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
// MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
// IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
// CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
// TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
// SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE
// ------
//
// Changes by Isaac Z. Schlueter released under the terms found in the
// LICENSE file within this project.
/**
* Integer greater than 0, representing some number of milliseconds, or the
* time at which a TTL started counting from.
*/
declare type LRUMilliseconds = number
/**
* An integer greater than 0, reflecting the calculated size of items
*/
declare type LRUSize = number
/**
* An integer greater than 0, reflecting a number of items
*/
declare type LRUCount = number
declare class LRUCache<K, V> implements Iterable<[K, V]> {
constructor(options: LRUCache.Options<K, V>)
/**
* Number of items in the cache.
* Alias for {@link size}
*
* @deprecated since 7.0 use {@link size} instead
*/
public readonly length: LRUCount
public readonly max: LRUCount
public readonly maxSize: LRUSize
public readonly maxEntrySize: LRUSize
public readonly sizeCalculation:
| LRUCache.SizeCalculator<K, V>
| undefined
public readonly dispose: LRUCache.Disposer<K, V>
/**
* @since 7.4.0
*/
public readonly disposeAfter: LRUCache.Disposer<K, V> | null
public readonly noDisposeOnSet: boolean
public readonly ttl: LRUMilliseconds
public readonly ttlResolution: LRUMilliseconds
public readonly ttlAutopurge: boolean
public readonly allowStale: boolean
public readonly updateAgeOnGet: boolean
/**
* @since 7.11.0
*/
public readonly noDeleteOnStaleGet: boolean
/**
* @since 7.6.0
*/
public readonly fetchMethod: LRUCache.Fetcher<K, V> | null
/**
* The total number of items held in the cache at the current moment.
*/
public readonly size: LRUCount
/**
* The total size of items in cache when using size tracking.
*/
public readonly calculatedSize: LRUSize
/**
* Add a value to the cache.
*/
public set(
key: K,
value: V,
options?: LRUCache.SetOptions<K, V>
): this
/**
* Return a value from the cache. Will update the recency of the cache entry
* found.
*
* If the key is not found, {@link get} will return `undefined`. This can be
* confusing when setting values specifically to `undefined`, as in
* `cache.set(key, undefined)`. Use {@link has} to determine whether a key is
* present in the cache at all.
*/
public get(key: K, options?: LRUCache.GetOptions<V>): V | undefined
/**
* Like {@link get} but doesn't update recency or delete stale items.
* Returns `undefined` if the item is stale, unless {@link allowStale} is set
* either on the cache or in the options object.
*/
public peek(key: K, options?: LRUCache.PeekOptions): V | undefined
/**
* Check if a key is in the cache, without updating the recency of use.
* Will return false if the item is stale, even though it is technically
* in the cache.
*
* Will not update item age unless {@link updateAgeOnHas} is set in the
* options or constructor.
*/
public has(key: K, options?: LRUCache.HasOptions<V>): boolean
/**
* Deletes a key out of the cache.
* Returns true if the key was deleted, false otherwise.
*/
public delete(key: K): boolean
/**
* Clear the cache entirely, throwing away all values.
*/
public clear(): void
/**
* Delete any stale entries. Returns true if anything was removed, false
* otherwise.
*/
public purgeStale(): boolean
/**
* Find a value for which the supplied fn method returns a truthy value,
* similar to Array.find(). fn is called as fn(value, key, cache).
*/
public find(
callbackFn: (
value: V,
key: K,
cache: this
) => boolean | undefined | void,
options?: LRUCache.GetOptions<V>
): V | undefined
/**
* Call the supplied function on each item in the cache, in order from
* most recently used to least recently used. fn is called as
* fn(value, key, cache). Does not update age or recenty of use.
*/
public forEach<T = this>(
callbackFn: (this: T, value: V, key: K, cache: this) => void,
thisArg?: T
): void
/**
* The same as {@link forEach} but items are iterated over in reverse
* order. (ie, less recently used items are iterated over first.)
*/
public rforEach<T = this>(
callbackFn: (this: T, value: V, key: K, cache: this) => void,
thisArg?: T
): void
/**
* Return a generator yielding the keys in the cache,
* in order from most recently used to least recently used.
*/
public keys(): Generator<K, void, void>
/**
* Inverse order version of {@link keys}
*
* Return a generator yielding the keys in the cache,
* in order from least recently used to most recently used.
*/
public rkeys(): Generator<K, void, void>
/**
* Return a generator yielding the values in the cache,
* in order from most recently used to least recently used.
*/
public values(): Generator<V, void, void>
/**
* Inverse order version of {@link values}
*
* Return a generator yielding the values in the cache,
* in order from least recently used to most recently used.
*/
public rvalues(): Generator<V, void, void>
/**
* Return a generator yielding `[key, value]` pairs,
* in order from most recently used to least recently used.
*/
public entries(): Generator<[K, V], void, void>
/**
* Inverse order version of {@link entries}
*
* Return a generator yielding `[key, value]` pairs,
* in order from least recently used to most recently used.
*/
public rentries(): Generator<[K, V], void, void>
/**
* Iterating over the cache itself yields the same results as
* {@link entries}
*/
public [Symbol.iterator](): Generator<[K, V], void, void>
/**
* Return an array of [key, entry] objects which can be passed to
* cache.load()
*/
public dump(): Array<[K, LRUCache.Entry<V>]>
/**
* Reset the cache and load in the items in entries in the order listed.
* Note that the shape of the resulting cache may be different if the
* same options are not used in both caches.
*/
public load(
cacheEntries: ReadonlyArray<[K, LRUCache.Entry<V>]>
): void
/**
* Evict the least recently used item, returning its value or `undefined`
* if cache is empty.
*/
public pop(): V | undefined
/**
* Deletes a key out of the cache.
*
* @deprecated since 7.0 use delete() instead
*/
public del(key: K): boolean
/**
* Clear the cache entirely, throwing away all values.
*
* @deprecated since 7.0 use clear() instead
*/
public reset(): void
/**
* Manually iterates over the entire cache proactively pruning old entries.
*
* @deprecated since 7.0 use purgeStale() instead
*/
public prune(): boolean
/**
* Make an asynchronous cached fetch using the {@link fetchMethod} function.
*
* If multiple fetches for the same key are issued, then they will all be
* coalesced into a single call to fetchMethod.
*
* Note that this means that handling options such as
* {@link allowStaleOnFetchAbort}, {@link signal}, and
* {@link allowStaleOnFetchRejection} will be determined by the FIRST fetch()
* call for a given key.
*
* This is a known (fixable) shortcoming which will be addresed on when
* someone complains about it, as the fix would involve added complexity and
* may not be worth the costs for this edge case.
*
* since: 7.6.0
*/
public fetch(
key: K,
options?: LRUCache.FetchOptions<K, V>
): Promise<V>
/**
* since: 7.6.0
*/
public getRemainingTTL(key: K): LRUMilliseconds
}
declare namespace LRUCache {
type DisposeReason = 'evict' | 'set' | 'delete'
type SizeCalculator<K, V> = (value: V, key: K) => LRUSize
type Disposer<K, V> = (
value: V,
key: K,
reason: DisposeReason
) => void
type Fetcher<K, V> = (
key: K,
staleValue: V | undefined,
options: FetcherOptions<K, V>
) => Promise<V | void | undefined> | V | void | undefined
interface DeprecatedOptions<K, V> {
/**
* alias for ttl
*
* @deprecated since 7.0 use options.ttl instead
*/
maxAge?: LRUMilliseconds
/**
* alias for {@link sizeCalculation}
*
* @deprecated since 7.0 use {@link sizeCalculation} instead
*/
length?: SizeCalculator<K, V>
/**
* alias for allowStale
*
* @deprecated since 7.0 use options.allowStale instead
*/
stale?: boolean
}
interface LimitedByCount {
/**
* The number of most recently used items to keep.
* Note that we may store fewer items than this if maxSize is hit.
*/
max: LRUCount
}
type MaybeMaxEntrySizeLimit<K, V> =
| {
/**
* The maximum allowed size for any single item in the cache.
*
* If a larger item is passed to {@link set} or returned by a
* {@link fetchMethod}, then it will not be stored in the cache.
*/
maxEntrySize: LRUSize
sizeCalculation?: SizeCalculator<K, V>
}
| {}
interface LimitedBySize<K, V> {
/**
* If you wish to track item size, you must provide a maxSize
* note that we still will only keep up to max *actual items*,
* if max is set, so size tracking may cause fewer than max items
* to be stored. At the extreme, a single item of maxSize size
* will cause everything else in the cache to be dropped when it
* is added. Use with caution!
*
* Note also that size tracking can negatively impact performance,
* though for most cases, only minimally.
*/
maxSize: LRUSize
/**
* Function to calculate size of items. Useful if storing strings or
* buffers or other items where memory size depends on the object itself.
*
* Items larger than {@link maxEntrySize} will not be stored in the cache.
*
* Note that when {@link maxSize} or {@link maxEntrySize} are set, every
* item added MUST have a size specified, either via a `sizeCalculation` in
* the constructor, or `sizeCalculation` or {@link size} options to
* {@link set}.
*/
sizeCalculation?: SizeCalculator<K, V>
}
interface LimitedByTTL {
/**
* Max time in milliseconds for items to live in cache before they are
* considered stale. Note that stale items are NOT preemptively removed
* by default, and MAY live in the cache, contributing to its LRU max,
* long after they have expired.
*
* Also, as this cache is optimized for LRU/MRU operations, some of
* the staleness/TTL checks will reduce performance, as they will incur
* overhead by deleting items.
*
* Must be an integer number of ms, defaults to 0, which means "no TTL"
*/
ttl: LRUMilliseconds
/**
* Boolean flag to tell the cache to not update the TTL when
* setting a new value for an existing key (ie, when updating a value
* rather than inserting a new value). Note that the TTL value is
* _always_ set (if provided) when adding a new entry into the cache.
*
* @default false
* @since 7.4.0
*/
noUpdateTTL?: boolean
/**
* Minimum amount of time in ms in which to check for staleness.
* Defaults to 1, which means that the current time is checked
* at most once per millisecond.
*
* Set to 0 to check the current time every time staleness is tested.
* (This reduces performance, and is theoretically unnecessary.)
*
* Setting this to a higher value will improve performance somewhat
* while using ttl tracking, albeit at the expense of keeping stale
* items around a bit longer than their TTLs would indicate.
*
* @default 1
* @since 7.1.0
*/
ttlResolution?: LRUMilliseconds
/**
* Preemptively remove stale items from the cache.
* Note that this may significantly degrade performance,
* especially if the cache is storing a large number of items.
* It is almost always best to just leave the stale items in
* the cache, and let them fall out as new items are added.
*
* Note that this means that {@link allowStale} is a bit pointless,
* as stale items will be deleted almost as soon as they expire.
*
* Use with caution!
*
* @default false
* @since 7.1.0
*/
ttlAutopurge?: boolean
/**
* Return stale items from {@link get} before disposing of them.
* Return stale values from {@link fetch} while performing a call
* to the {@link fetchMethod} in the background.
*
* @default false
*/
allowStale?: boolean
/**
* Update the age of items on {@link get}, renewing their TTL
*
* @default false
*/
updateAgeOnGet?: boolean
/**
* Do not delete stale items when they are retrieved with {@link get}.
* Note that the {@link get} return value will still be `undefined` unless
* allowStale is true.
*
* @default false
* @since 7.11.0
*/
noDeleteOnStaleGet?: boolean
/**
* Update the age of items on {@link has}, renewing their TTL
*
* @default false
*/
updateAgeOnHas?: boolean
}
type SafetyBounds<K, V> =
| LimitedByCount
| LimitedBySize<K, V>
| LimitedByTTL
// options shared by all three of the limiting scenarios
interface SharedOptions<K, V> {
/**
* Function that is called on items when they are dropped from the cache.
* This can be handy if you want to close file descriptors or do other
* cleanup tasks when items are no longer accessible. Called with `key,
* value`. It's called before actually removing the item from the
* internal cache, so it is *NOT* safe to re-add them.
* Use {@link disposeAfter} if you wish to dispose items after they have
* been full removed, when it is safe to add them back to the cache.
*/
dispose?: Disposer<K, V>
/**
* The same as dispose, but called *after* the entry is completely
* removed and the cache is once again in a clean state. It is safe to
* add an item right back into the cache at this point.
* However, note that it is *very* easy to inadvertently create infinite
* recursion this way.
*
* @since 7.3.0
*/
disposeAfter?: Disposer<K, V>
/**
* Set to true to suppress calling the dispose() function if the entry
* key is still accessible within the cache.
* This may be overridden by passing an options object to {@link set}.
*
* @default false
*/
noDisposeOnSet?: boolean
/**
* Function that is used to make background asynchronous fetches. Called
* with `fetchMethod(key, staleValue, { signal, options, context })`.
*
* If `fetchMethod` is not provided, then {@link fetch} is
* equivalent to `Promise.resolve(cache.get(key))`.
*
* The `fetchMethod` should ONLY return `undefined` in cases where the
* abort controller has sent an abort signal.
*
* @since 7.6.0
*/
fetchMethod?: LRUCache.Fetcher<K, V>
/**
* Set to true to suppress the deletion of stale data when a
* {@link fetchMethod} throws an error or returns a rejected promise
*
* This may be overridden in the {@link fetchMethod}.
*
* @default false
* @since 7.10.0
*/
noDeleteOnFetchRejection?: boolean
/**
* Set to true to allow returning stale data when a {@link fetchMethod}
* throws an error or returns a rejected promise. Note that this
* differs from using {@link allowStale} in that stale data will
* ONLY be returned in the case that the fetch fails, not any other
* times.
*
* This may be overridden in the {@link fetchMethod}.
*
* @default false
* @since 7.16.0
*/
allowStaleOnFetchRejection?: boolean
/**
*
* Set to true to ignore the `abort` event emitted by the `AbortSignal`
* object passed to {@link fetchMethod}, and still cache the
* resulting resolution value, as long as it is not `undefined`.
*
* When used on its own, this means aborted {@link fetch} calls are not
* immediately resolved or rejected when they are aborted, and instead take
* the full time to await.
*
* When used with {@link allowStaleOnFetchAbort}, aborted {@link fetch}
* calls will resolve immediately to their stale cached value or
* `undefined`, and will continue to process and eventually update the
* cache when they resolve, as long as the resulting value is not
* `undefined`, thus supporting a "return stale on timeout while
* refreshing" mechanism by passing `AbortSignal.timeout(n)` as the signal.
*
* **Note**: regardless of this setting, an `abort` event _is still emitted
* on the `AbortSignal` object_, so may result in invalid results when
* passed to other underlying APIs that use AbortSignals.
*
* This may be overridden in the {@link fetchMethod} or the call to
* {@link fetch}.
*
* @default false
* @since 7.17.0
*/
ignoreFetchAbort?: boolean
/**
* Set to true to return a stale value from the cache when the
* `AbortSignal` passed to the {@link fetchMethod} dispatches an `'abort'`
* event, whether user-triggered, or due to internal cache behavior.
*
* Unless {@link ignoreFetchAbort} is also set, the underlying
* {@link fetchMethod} will still be considered canceled, and its return
* value will be ignored and not cached.
*
* This may be overridden in the {@link fetchMethod} or the call to
* {@link fetch}.
*
* @default false
* @since 7.17.0
*/
allowStaleOnFetchAbort?: boolean
/**
* Set to any value in the constructor or {@link fetch} options to
* pass arbitrary data to the {@link fetchMethod} in the {@link context}
* options field.
*
* @since 7.12.0
*/
fetchContext?: any
}
type Options<K, V> = SharedOptions<K, V> &
DeprecatedOptions<K, V> &
SafetyBounds<K, V> &
MaybeMaxEntrySizeLimit<K, V>
/**
* options which override the options set in the LRUCache constructor
* when making calling {@link set}.
*/
interface SetOptions<K, V> {
/**
* A value for the size of the entry, prevents calls to
* {@link sizeCalculation}.
*
* Items larger than {@link maxEntrySize} will not be stored in the cache.
*
* Note that when {@link maxSize} or {@link maxEntrySize} are set, every
* item added MUST have a size specified, either via a `sizeCalculation` in
* the constructor, or {@link sizeCalculation} or `size` options to
* {@link set}.
*/
size?: LRUSize
/**
* Overrides the {@link sizeCalculation} method set in the constructor.
*
* Items larger than {@link maxEntrySize} will not be stored in the cache.
*
* Note that when {@link maxSize} or {@link maxEntrySize} are set, every
* item added MUST have a size specified, either via a `sizeCalculation` in
* the constructor, or `sizeCalculation` or {@link size} options to
* {@link set}.
*/
sizeCalculation?: SizeCalculator<K, V>
ttl?: LRUMilliseconds
start?: LRUMilliseconds
noDisposeOnSet?: boolean
noUpdateTTL?: boolean
status?: Status<V>
}
/**
* options which override the options set in the LRUCAche constructor
* when calling {@link has}.
*/
interface HasOptions<V> {
updateAgeOnHas?: boolean
status: Status<V>
}
/**
* options which override the options set in the LRUCache constructor
* when calling {@link get}.
*/
interface GetOptions<V> {
allowStale?: boolean
updateAgeOnGet?: boolean
noDeleteOnStaleGet?: boolean
status?: Status<V>
}
/**
* options which override the options set in the LRUCache constructor
* when calling {@link peek}.
*/
interface PeekOptions {
allowStale?: boolean
}
/**
* Options object passed to the {@link fetchMethod}
*
* May be mutated by the {@link fetchMethod} to affect the behavior of the
* resulting {@link set} operation on resolution, or in the case of
* {@link noDeleteOnFetchRejection}, {@link ignoreFetchAbort}, and
* {@link allowStaleOnFetchRejection}, the handling of failure.
*/
interface FetcherFetchOptions<K, V> {
allowStale?: boolean
updateAgeOnGet?: boolean
noDeleteOnStaleGet?: boolean
size?: LRUSize
sizeCalculation?: SizeCalculator<K, V>
ttl?: LRUMilliseconds
noDisposeOnSet?: boolean
noUpdateTTL?: boolean
noDeleteOnFetchRejection?: boolean
allowStaleOnFetchRejection?: boolean
ignoreFetchAbort?: boolean
allowStaleOnFetchAbort?: boolean
status?: Status<V>
}
/**
* Status object that may be passed to {@link fetch}, {@link get},
* {@link set}, and {@link has}.
*/
interface Status<V> {
/**
* The status of a set() operation.
*
* - add: the item was not found in the cache, and was added
* - update: the item was in the cache, with the same value provided
* - replace: the item was in the cache, and replaced
* - miss: the item was not added to the cache for some reason
*/
set?: 'add' | 'update' | 'replace' | 'miss'
/**
* the ttl stored for the item, or undefined if ttls are not used.
*/
ttl?: LRUMilliseconds
/**
* the start time for the item, or undefined if ttls are not used.
*/
start?: LRUMilliseconds
/**
* The timestamp used for TTL calculation
*/
now?: LRUMilliseconds
/**
* the remaining ttl for the item, or undefined if ttls are not used.
*/
remainingTTL?: LRUMilliseconds
/**
* The calculated size for the item, if sizes are used.
*/
size?: LRUSize
/**
* A flag indicating that the item was not stored, due to exceeding the
* {@link maxEntrySize}
*/
maxEntrySizeExceeded?: true
/**
* The old value, specified in the case of `set:'update'` or
* `set:'replace'`
*/
oldValue?: V
/**
* The results of a {@link has} operation
*
* - hit: the item was found in the cache
* - stale: the item was found in the cache, but is stale
* - miss: the item was not found in the cache
*/
has?: 'hit' | 'stale' | 'miss'
/**
* The status of a {@link fetch} operation.
* Note that this can change as the underlying fetch() moves through
* various states.
*
* - inflight: there is another fetch() for this key which is in process
* - get: there is no fetchMethod, so {@link get} was called.
* - miss: the item is not in cache, and will be fetched.
* - hit: the item is in the cache, and was resolved immediately.
* - stale: the item is in the cache, but stale.
* - refresh: the item is in the cache, and not stale, but
* {@link forceRefresh} was specified.
*/
fetch?: 'get' | 'inflight' | 'miss' | 'hit' | 'stale' | 'refresh'
/**
* The {@link fetchMethod} was called
*/
fetchDispatched?: true
/**
* The cached value was updated after a successful call to fetchMethod
*/
fetchUpdated?: true
/**
* The reason for a fetch() rejection. Either the error raised by the
* {@link fetchMethod}, or the reason for an AbortSignal.
*/
fetchError?: Error
/**
* The fetch received an abort signal
*/
fetchAborted?: true
/**
* The abort signal received was ignored, and the fetch was allowed to
* continue.
*/
fetchAbortIgnored?: true
/**
* The fetchMethod promise resolved successfully
*/
fetchResolved?: true
/**
* The fetchMethod promise was rejected
*/
fetchRejected?: true
/**
* The status of a {@link get} operation.
*
* - fetching: The item is currently being fetched. If a previous value is
* present and allowed, that will be returned.
* - stale: The item is in the cache, and is stale.
* - hit: the item is in the cache
* - miss: the item is not in the cache
*/
get?: 'stale' | 'hit' | 'miss'
/**
* A fetch or get operation returned a stale value.
*/
returnedStale?: true
}
/**
* options which override the options set in the LRUCache constructor
* when calling {@link fetch}.
*
* This is the union of GetOptions and SetOptions, plus
* {@link noDeleteOnFetchRejection}, {@link allowStaleOnFetchRejection},
* {@link forceRefresh}, and {@link fetchContext}
*/
interface FetchOptions<K, V> extends FetcherFetchOptions<K, V> {
forceRefresh?: boolean
fetchContext?: any
signal?: AbortSignal
status?: Status<V>
}
interface FetcherOptions<K, V> {
signal: AbortSignal
options: FetcherFetchOptions<K, V>
/**
* Object provided in the {@link fetchContext} option
*/
context: any
}
interface Entry<V> {
value: V
ttl?: LRUMilliseconds
size?: LRUSize
start?: LRUMilliseconds
}
}
export = LRUCache

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,96 @@
{
"name": "lru-cache",
"description": "A cache object that deletes the least-recently-used items.",
"version": "7.18.3",
"author": "Isaac Z. Schlueter <i@izs.me>",
"keywords": [
"mru",
"lru",
"cache"
],
"sideEffects": false,
"scripts": {
"build": "npm run prepare",
"pretest": "npm run prepare",
"presnap": "npm run prepare",
"prepare": "node ./scripts/transpile-to-esm.js",
"size": "size-limit",
"test": "tap",
"snap": "tap",
"preversion": "npm test",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"format": "prettier --write .",
"typedoc": "typedoc ./index.d.ts"
},
"type": "commonjs",
"main": "./index.js",
"module": "./index.mjs",
"types": "./index.d.ts",
"exports": {
".": {
"import": {
"types": "./index.d.ts",
"default": "./index.mjs"
},
"require": {
"types": "./index.d.ts",
"default": "./index.js"
}
},
"./package.json": "./package.json"
},
"repository": "git://github.com/isaacs/node-lru-cache.git",
"devDependencies": {
"@size-limit/preset-small-lib": "^7.0.8",
"@types/node": "^17.0.31",
"@types/tap": "^15.0.6",
"benchmark": "^2.1.4",
"c8": "^7.11.2",
"clock-mock": "^1.0.6",
"eslint-config-prettier": "^8.5.0",
"prettier": "^2.6.2",
"size-limit": "^7.0.8",
"tap": "^16.3.4",
"ts-node": "^10.7.0",
"tslib": "^2.4.0",
"typedoc": "^0.23.24",
"typescript": "^4.6.4"
},
"license": "ISC",
"files": [
"index.js",
"index.mjs",
"index.d.ts"
],
"engines": {
"node": ">=12"
},
"prettier": {
"semi": false,
"printWidth": 70,
"tabWidth": 2,
"useTabs": false,
"singleQuote": true,
"jsxSingleQuote": false,
"bracketSameLine": true,
"arrowParens": "avoid",
"endOfLine": "lf"
},
"tap": {
"nyc-arg": [
"--include=index.js"
],
"node-arg": [
"--expose-gc",
"--require",
"ts-node/register"
],
"ts": false
},
"size-limit": [
{
"path": "./index.js"
}
]
}

View File

@@ -0,0 +1,79 @@
{
"name": "make-fetch-happen",
"version": "10.2.1",
"description": "Opinionated, caching, retrying fetch client",
"main": "lib/index.js",
"files": [
"bin/",
"lib/"
],
"scripts": {
"preversion": "npm test",
"postversion": "npm publish",
"prepublishOnly": "git push origin --follow-tags",
"test": "tap",
"posttest": "npm run lint",
"eslint": "eslint",
"lint": "eslint \"**/*.js\"",
"lintfix": "npm run lint -- --fix",
"postlint": "template-oss-check",
"snap": "tap",
"template-oss-apply": "template-oss-apply --force"
},
"repository": {
"type": "git",
"url": "https://github.com/npm/make-fetch-happen.git"
},
"keywords": [
"http",
"request",
"fetch",
"mean girls",
"caching",
"cache",
"subresource integrity"
],
"author": "GitHub Inc.",
"license": "ISC",
"dependencies": {
"agentkeepalive": "^4.2.1",
"cacache": "^16.1.0",
"http-cache-semantics": "^4.1.0",
"http-proxy-agent": "^5.0.0",
"https-proxy-agent": "^5.0.0",
"is-lambda": "^1.0.1",
"lru-cache": "^7.7.1",
"minipass": "^3.1.6",
"minipass-collect": "^1.0.2",
"minipass-fetch": "^2.0.3",
"minipass-flush": "^1.0.5",
"minipass-pipeline": "^1.2.4",
"negotiator": "^0.6.3",
"promise-retry": "^2.0.1",
"socks-proxy-agent": "^7.0.0",
"ssri": "^9.0.0"
},
"devDependencies": {
"@npmcli/eslint-config": "^3.0.1",
"@npmcli/template-oss": "3.5.0",
"mkdirp": "^1.0.4",
"nock": "^13.2.4",
"rimraf": "^3.0.2",
"safe-buffer": "^5.2.1",
"standard-version": "^9.3.2",
"tap": "^16.0.0"
},
"engines": {
"node": "^12.13.0 || ^14.15.0 || >=16.0.0"
},
"tap": {
"color": 1,
"files": "test/*.js",
"check-coverage": true,
"timeout": 60
},
"templateOSS": {
"//@npmcli/template-oss": "This file is partially managed by @npmcli/template-oss. Edits may be overwritten.",
"version": "3.5.0"
}
}