Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization: use the 'params' option for the script processor #372

Open
joegallo opened this issue Jun 5, 2023 · 1 comment
Open

Optimization: use the 'params' option for the script processor #372

joegallo opened this issue Jun 5, 2023 · 1 comment

Comments

@joegallo
Copy link

joegallo commented Jun 5, 2023

A small (but real!) optimization would be to extract the 'typeMap' from this script and use the params option of the script processor to pass that same information to the script instead:

- script:
ignore_failure: true
if: "ctx.network?.protocol == 'dns' && ctx.dns?.question?.Ext_temp?.type != null && ctx.dns?.question?.type == null"
# the parsed type is a string of a number, so we'll want to convert it to the resource reference format
# question.type map references:
# https://github.com/spc476/SPCDNS/blob/master/src/dns.h
# https://pkg.go.dev/github.com/miekg/dns#pkg-constants
# https://en.wikipedia.org/wiki/List_of_DNS_record_types
source: >-
Map typeMap = ['1': 'A', '2': 'NS', '3': 'MD', '4': 'MF', '5': 'CNAME', '6': 'SOA', '7': 'MB', '8': 'MG', '9': 'MR',
'10': 'NULL', '11': 'WKS', '12': 'PTR', '13': 'HINFO', '14': 'MINFO', '15': 'MX', '16': 'TXT', '17': 'RP', '18': 'AFSDB',
'19': 'X25', '20': 'ISDN', '21': 'RT', '22': 'NSAP', '23': 'NSAPPTR', '24': 'SIG', '25': 'KEY', '26': 'PX',
'27': 'GPOS', '28': 'AAAA', '29': 'LOC', '30': 'NXT', '31': 'EID', '32': 'NIMLOC', '33': 'SRV', '34': 'ATMA',
'35': 'NAPTR', '36': 'KX', '37': 'CERT', '38': 'A6', '39': 'DNAME', '40': 'SINK', '41': 'OPT', '42': 'APL',
'43': 'DS', '44': 'SSHFP', '45': 'ISECKEY', '46': 'RRSIG', '47': 'NSEC', '48': 'DNSKEY', '49': 'DHCID',
'50': 'NSEC3', '51': 'NSEC3PARAM', '52': 'TLSA', '53': 'SMIMEA', '55': 'HIP', '56': 'NINFO', '57': 'RKEY',
'58': 'TALINK', '59': 'CDS', '60': 'CDNSKEY', '61': 'OPENPGPKEY', '62': 'CSYNC', '63': 'ZONEMD', '64': 'SVCB',
'65': 'HTTPS', '99': 'SPF', '100': 'UINFO', '101': 'UID', '102': 'GID', '103': 'UNSPEC', '104': 'NID',
'105': 'L32', '106': 'L64', '107': 'LP', '108': 'EUI48', '109': 'EUI64', '249': 'TKEY', '250': 'TSIG',
'251': 'IXFR', '252': 'AXFR', '253': 'MAILB', '254': 'MAILA', '255': 'ANY','256': 'URI', '257': 'CAA',
'258': 'AVC'];
def type = typeMap[ctx.dns.question.Ext_temp.type];
if (type != null) {
ctx.dns.question.type = type;
}

As written, the script processor builds that map from scratch every time for every document, but if you switch to the params pattern, then the map would be constructed just once and passed into the script processor for every document.

/cc @pzl


I noticed this while processing flamegraphs to see where this pipeline was spending its time, you can see here that the vast majority of the time spent by this script processor is just in building up the map for processing, rather than actually doing the real work it's there to do:

bad map every invocation 2

edit: Note, however, it's still just a fraction of the time spent in any processor at all. While this change will speed up the script processor in question quite a bit, I don't want you to think it's going to improve the performance of the whole pipeline enormously (but every couple of percents count, right?).

@joegallo
Copy link
Author

joegallo commented Jun 5, 2023

elastic/integrations#3492 is an example of a similar change to a different script processor for another one of our integrations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant