Typescript and types

[chris74a]

Hello,
I am trying to use node-crawler in typescript. I do npm install @types/crawler , but I have a 404.
Is there a way to use this module with ts?
Thanks

[mike442144]

Actually I'm not familiar with ts, contribute is welcome. :)

[raynor85]

I have added a index.d.ts in my project with the following basic type definitions. It is not exhaustive, but it is a start.

declare module 'crawler' {
  import { IncomingMessage } from 'http';

  export interface CrawlerResponse extends IncomingMessage {
    $: CheerioStatic;
    options: CrawlerOptions;
    request: any;
  }

  export type CrawlerCallback = (
    error: Error,
    res: CrawlerResponse,
    done: () => {}
  ) => void;

  interface JQueryObject {
    name: string;
    options: {
      normalizeWhitespace?: boolean;
      xmlMode?: boolean;
      decodeEntities?: boolean;
    };
  }

  type JQuery = boolean | 'cheerio' | JQueryObject;

  export interface CrawlerOptions {
    uri?: string;
    autoWindowClose?: boolean;
    forceUTF8?: boolean;
    gzip?: boolean;
    incomingEncoding?: string;
    jQuery?: JQuery;
    maxConnections?: number;
    method?: string;
    priority?: number;
    priorityRange?: number;
    rateLimit?: number;
    referer?: boolean;
    retries?: number;
    retryTimeout?: number;
    timeout?: number;
    skipDuplicates?: boolean;
    rotateUA?: boolean;
    homogeneous?: boolean;
    callback?: CrawlerCallback;
  }

  class Crawler {
    constructor(options: CrawlerOptions);
    queue(options: string | string[] | CrawlerOptions | CrawlerOptions[]): void;
  }

  export default Crawler;
}

[mike442144]

Though I don't know much about ts, I think it is a good start :)

[fanhuahaitang]

import request from 'request';
import events from 'events';

export interface CreateCrawlerOptions {
autoWindowClose?: boolean;
forceUTF8?: boolean;
gzip?: boolean;
incomingEncoding?: string;
jquery?: boolean | string | CheerioOptionsInterface | any;
jQuery?: boolean | string | CheerioOptionsInterface | any;
maxConnections?: number;
method?: string;
priority?: number;
priorityRange?: number;
rateLimit?: number;
referer?: false | string
retries?: number;
retryTimeout?: number;
timeout?: number;
skipDuplicates?: boolean;
rotateUA?: boolean;
userAgent?: string | string[];
homogeneous?: boolean;
debug?: boolean;
logger?: {
log: (level: string, ...args: any[]) => void;
};
seenreq?: any;
headers?: request.Headers;
callback?: (err: Error, res: CrawlerRequestResponse, done: () => void) => void;
[x: string]: any
}

export declare class Crawler extends events.EventEmitter {
queueSize: number;
on(channel: 'schedule', listener: (options: CrawlerRequestOptions) => void): this;
on(channel: 'limiterChange', listener: (options: CrawlerRequestOptions, limiter: string) => void): this;
on(channel: 'request', listener: (options: CrawlerRequestOptions) => void): this;
on(channel: 'drain', listener: () => void): this;
queue(uri: string): void;
queue(options: CrawlerRequestOptions): void;
direct(options: CrawlerRequestOptions & {
callback: (error: Error, response: CrawlerRequestResponse) => void;
}): void;
setLimiterProperty(limiter: string, property?: string, value?: any): void;
}

export interface CrawlerRequestOptions extends request.CoreOptions {
uri?: string;
html?: string;
proxy?: any;
proxies?: any[];
limiter?: string;
encoding?: string;
priority?: number;
jQuery?: boolean | string | CheerioOptionsInterface | any;
preRequest?: (options: CrawlerRequestOptions, doRequest: (err: Error) => void) => void;
callback?: (err: Error, res: CrawlerRequestResponse, done: () => void) => void;
[x: string]: any
}

export interface CrawlerRequestResponse {
statusMessage?: string;
statusCode: number;
body?: Buffer | string;
headers?: request.Headers;
request?: request.RequestAsJSON;
options?: CrawlerRequestOptions;
$?: CheerioAPI
[x: string]: any
}

[fanhuahaitang]

The code is too complicated,and it's packages of Bottleneck/seenreq/whacko don't support ts.

declare module 'crawler' {
  import { IncomingMessage } from 'http';

  import request from 'request';
  import events from 'events';

  class Crawler extends events.EventEmitter {
    constructor(options: CreateCrawlerOptions);
    queueSize: number;
    on(channel: 'schedule', listener: (options: CrawlerRequestOptions) => void): this;
    on(channel: 'limiterChange', listener: (options: CrawlerRequestOptions, limiter: string) => void): this;
    on(channel: 'request', listener: (options: CrawlerRequestOptions) => void): this;
    on(channel: 'drain', listener: () => void): this;
    queue(uri: string): void;
    queue(uri: string[]): void;
    queue(options: CrawlerRequestOptions | CrawlerRequestOptions[]): void;
    direct(options: CrawlerRequestOptions & {
      callback: (error: Error, response: CrawlerRequestResponse) => void;
    }): void;
    setLimiterProperty(limiter: string, property?: string, value?: any): void;
  }

  export interface CrawlerRequestOptions extends request.CoreOptions {
    uri?: string;
    html?: string;
    proxy?: any;
    proxies?: any[];
    limiter?: string;
    encoding?: string;
    priority?: number;
    jQuery?: boolean | "whacko" | "cheerio" | {
        name: "cheerio";
        options: CheerioOptionsInterface;
      } | {
        jsdom: any;
        [key: string]: any;
      } | any;
    preRequest?: (options: CrawlerRequestOptions, doRequest: (err: Error) => void) => void;
    callback?: (err: Error, res: CrawlerRequestResponse, done: () => void) => void;
    [x: string]: any
  }

  export interface CrawlerRequestResponse extends IncomingMessage {
    body?: Buffer | string;
    request?: request.RequestAsJSON;
    options?: CrawlerRequestOptions;
    $?: CheerioAPI
    [x: string]: any
  }

  export interface CreateCrawlerOptions {
    autoWindowClose?: boolean;
    forceUTF8?: boolean;
    gzip?: boolean;
    incomingEncoding?: string;
    jquery?: boolean | "whacko" | "cheerio" | {
        name: "cheerio";
        options: CheerioOptionsInterface;
      } | {
        jsdom: any;
        [key: string]: any;
      } | any;
    jQuery?: boolean | "whacko" | "cheerio" | {
        name: "cheerio";
        options: CheerioOptionsInterface;
      } | {
        jsdom: any;
        [key: string]: any;
      } | any;
    maxConnections?: number;
    method?: string;
    priority?: number;
    priorityRange?: number;
    rateLimit?: number;
    referer?: false | string;
    retries?: number;
    retryTimeout?: number;
    timeout?: number;
    skipDuplicates?: boolean;
    rotateUA?: boolean;
    userAgent?: string | string[];
    homogeneous?: boolean;
    debug?: boolean;
    logger?: {
      log: (level: string, ...args: any[]) => void;
    };
    seenreq?: any;
    headers?: request.Headers;
    callback?: (err: Error, res: CrawlerRequestResponse, done: () => void) => void;
    [x: string]: any;
  }

  export default Crawler;
}

[mike442144]

@raynor85 @wakhh thanks for your great job, I'm glad to merge your pull request.

[vczh]

Have this typescript description file been submitted to npm?

[mike442144]

@vczh Nope, I don't know much about Typescript. Are you going to add this feature with confidence?

[vczh]

@mike442144 maybe this is a good chance to learn, by rewriting the code to TypeScript, just need to add type annotation, don't need to change logic :)

The problem for me is that, I am not the author of this library, how could I know if the type description is actually correct.

[mike442144]

@vczh My issue is that I'm too busy to do this, so do not count on me. If you're using crawler, it is a good chance to do this to help others : ).

if you're not confident with the type definition, we can discuss in this issue.

[vczh]

@mike442144 I tried this library before with @wakhh 's work in this thread and it works fine. I only used a very small subset of the feature so I am not sure. It only need a little update to remove warnings.

I was trying to write a tool to download my own website that host locally on my computer, but I found this library doesn't read into HTML data and therefore not do recursive requests. Maybe I was wrong, but I cannot find related information from the document, and I turned to website-scraper. Unfortunately I've deleted my update to the type.

[mike442144]

@vczh I'm sure you missed something. crawler provides almost all the functions you need, but need some efforts to compose and organize. You're welcomed to come back : )

[fanhuahaitang]

I update the ts code directly on last reply and fix the error of jquery key.
But i don't use it now,i rewirite a crawler api lib to crawler file,and use npm promise-queue to control it.
https://www.npmjs.com/package/crawler-api