fixed bug and made ApiScout ready for PyPI

Author: Daniel Plohmann (jupiter)

.gitignore

@@ -63,3 +63,4 @@ ENV/
 # other
 dbs/*.csv
 config.py
+.pylintrc

Makefile

@@ -1,7 +1,18 @@
 init:
 	pip install -r requirements.txt
-
+package:
+	rm -rf dist/*
+	python3 setup.py sdist
+publish:
+	python3 -m twine upload dist/*
+pylint:
+	python3 -m pylint --rcfile=.pylintrc apiscout
 test:
-	nosetests tests
+	python3 -m nose
+test-coverage:
+	python3 -m nose --with-coverage --cover-erase --cover-html-dir=./coverage-html --cover-html --cover-package=apiscout
 clean:
 	find . | grep -E "(__pycache__|\.pyc|\.pyo$\)" | xargs rm -rf
+	rm -rf .coverage
+	rm -rf coverage-html
+	rm -rf dist/*

README.md

@@ -0,0 +1,53 @@
+# ApiScout
+
+This project aims at simplifying Windows API import recovery.
+As input, arbitrary memory dumps for a known environment can be processed (please note: a reference DB has to be built first, using apiscout/db_builder).  
+The output is an ordered list of identified Windows API references with some meta information, and an ApiVector fingerprint.  
+
+* scout.py -- should give a good outline on how to work with the library.  
+* ida_scout.py -- is a convenience GUI wrapper for use in IDA Pro.  
+* match.py -- demonstrates how ApiVectors can be matched against each other and collections of fingerprints.  
+* collect.py -- builds a database of WinAPI fingerprints (ApiVectors) that can be used for matching.  
+* export.py -- generates ApiQR diagrams that visualize ApiVectors.  
+* update.py -- pull the most recent ApiVector DB from Malpedia (requires Malpedia account / API token).  
+
+The code should be fully compatible with Python 2 and 3.  
+There is a blog post describing ApiScout in more detail: http://byte-atlas.blogspot.com/2017/04/apiscout.html.  
+Also, another blog post explaining how ApiVectors are constructed and stored: https://byte-atlas.blogspot.com/2018/04/apivectors.html.  
+We also presented a paper at Botconf 2018 that describes the ApiScout methodology in-depth, including an evaluation over Malpedia: https://journal.cecyf.fr/ojs/index.php/cybin/article/view/20/23  
+
+## Version History
+
+ * 2020-06-30: v1.1.0 - Now using LIEF for import table parsing. Fixed bug which would not produce ApiVectors when using import table parsing. ApiScout is now also available through PyPI.
+ * 2020-03-03: Added a script to pull the most recent ApiVector DB from Malpedia (requires Malpedia account / API token).
+ * 2020-03-02: Ported to IDA 7.4 (THX to @jenfrie).
+ * 2020-02-18: DB Builder is now compatible up to Python 3.7 (THX to @elanfer).
+ * 2019-10-08: Workaround for broken filtering of the API view in IDA 7.3 (THX to @enzok for pointing this out).
+ * 2019-08-22: Fixed a bug where missing type info in IDA would lead to a crash (now gives an error message instead).
+ * 2019-08-20: Added self-filter to eliminate pointers to own memory image that could be mistakenly treated as API references.
+ * 2019-06-06: Added support for proper type reconstruction for annotated APIs in IDA Pro (THX to @FlxP0c)
+ * 2019-05-15: Added numpy support for vector calculations (based on implementation provided by @garanews - THX!)
+ * 2019-05-15: Fixed a bug in PE mapper where buffer would be shortened because of misinterpretation of section sizes.
+ * 2019-01-23: QoL improvements: automated data folder deployment when used as module, logger initialization (THX to @jdval)
+ * 2018-08-23: Fixed a bug in PE mapper where the PE header would be overwritten by (empty) section data.
+ * 2018-08-21: Added functionality that allows to use import table information instead of crawling for references.
+ * 2018-07-31: Fixed convenience functions to create/export vectors from/to lists and dicts, added test coverage.
+ * 2018-07-23: WARNING: Change in Apivector format -- Introduced sorted ApiVectors which are even more space efficient (20%+).
+ * 2018-06-25: Fixed incompatibility with IDA Pro 7.0+ (THX to @nazywam!)
+ * 2018-05-23: Added further semantic context groups (THX to Quoscient.io)
+ * 2018-03-27: Heuristic estimation of Windows API reference counts added
+ * 2018-03-06: ApiQR visualization of vector results (C-1024)
+ * 2017-11-28: Added own import table parser to enrich result information
+ * 2017-08-24: Multi-Segment support in IDA Pro (THX to @nazywam!)
+ * 2017-05-31: Added Windows 7 SP1 64bit import DB (compatible to Malpedia)
+
+## Credits
+
+The idea has previously gone through multiple iterations until reaching this refactored release.  
+Thanks to Thorsten Jenke and Steffen Enders for their previous endeavours and evaluating a proof-of-concept of this method.  
+More thanks to Steffen Enders for his work on the visualization of ApiQR diagrams.  
+Also thanks to Ero Carrera for pefile and Elias Bachaalany for the IDA Python AskUsingForm template. :)  
+Additionally many thanks to Andrea Garavaglia for his performance benchmarks that lead to drastic speedups in the applied matching!  
+
+
+Pull requests welcome! :)

README.rst

@@ -30,6 +30,12 @@
 import operator
 import logging
 
+try:
+    import lief
+except:
+    print("lief is not installed! We recommend installing lief to improve import table parsing capabilities of ApiScout!")
+    lief = None
+
 from .ImportTableLoader import ImportTableLoader
 from .ApiVector import ApiVector
 from .PeTools import PeTools
@@ -54,6 +60,7 @@ def __init__(self, db_filepath=None):
         if db_filepath:
             self.loadDbFile(db_filepath)
         self._apivector = ApiVector()
+        self.loadWinApi1024()
 
     def loadDbFile(self, db_filepath):
         api_db = {}
@@ -88,7 +95,10 @@ def loadDbFile(self, db_filepath):
         LOG.info("loaded %d exports from %d DLLs (%s) with %d potential collisions.", num_apis_loaded, len(api_db["dlls"]), api_db["os_name"], num_collisions)
         self.api_maps[api_db["os_name"]] = api_map
 
-    def loadWinApi1024(self, winapi1024_filepath):
+    def loadWinApi1024(self, winapi1024_filepath=None):
+        if winapi1024_filepath is None:
+            this_dir = os.path.abspath(os.path.join(os.path.dirname(__file__)))
+            winapi1024_filepath =  this_dir + os.sep + "data" + os.sep + "winapi1024v1.txt"
         self._apivector = ApiVector(winapi1024_filepath)
 
     def _resolveApiByAddress(self, api_map_name, absolute_addr):
@@ -177,20 +187,30 @@ def _getCodeReferences(self, binary):
         return references
 
     def evaluateImportTable(self, binary, is_unmapped=True):
+        self._binary_length = len(binary)
         results = {"import_table": []}
-        mapped_binary = binary
-        if is_unmapped:
-            LOG.debug("Mapping unmapped binary before processing")
-            mapped_binary = PeTools.mapBinary(binary)
-        bitness = PeTools.getBitness(mapped_binary)
-        self._import_table = None
-        self._parseImportTable(mapped_binary)
-        references = self._getCodeReferences(mapped_binary)
-        for offset, import_entry in sorted(self._import_table.items()):
-            ref_count = 1
-            if bitness:
-                ref_count = 1 + references[bitness][offset] if offset in references[bitness] else 1
-            results["import_table"].append((offset + self.load_offset, 0, import_entry["dll_name"].lower() + "_0x0", import_entry["name"], bitness, True, ref_count))
+        if lief:
+            lief_binary = lief.parse(bytearray(binary))
+            bitness = 32 if lief_binary.header.machine == lief.PE.MACHINE_TYPES.I386 else 64
+            for imported_library in lief_binary.imports:
+                for func in imported_library.entries:
+                    if func.name:
+                        results["import_table"].append((func.iat_address + self.load_offset, 0xFFFFFFFF, imported_library.name.lower() + "_0x0", func.name, bitness, True, 0))
+        else:
+            # fallback using the old method and out own import table parser
+            mapped_binary = binary
+            if is_unmapped:
+                LOG.debug("Mapping unmapped binary before processing")
+                mapped_binary = PeTools.mapBinary(binary)
+            bitness = PeTools.getBitness(mapped_binary)
+            self._import_table = None
+            self._parseImportTable(mapped_binary)
+            references = self._getCodeReferences(mapped_binary)
+            for offset, import_entry in sorted(self._import_table.items()):
+                ref_count = 1
+                if bitness:
+                    ref_count = 1 + references[bitness][offset] if offset in references[bitness] else 1
+                results["import_table"].append((offset + self.load_offset, 0xFFFFFFFF, import_entry["dll_name"].lower() + "_0x0", import_entry["name"], bitness, True, ref_count))
         return results
 
     def crawl(self, binary):

apiscout/ApiScout.py

@@ -2,3 +2,4 @@ nose
 Pillow
 numpy
 requests
+lief

requirements.txt

@@ -44,10 +44,6 @@ def get_all_db_files():
     return [db_dir + fn for fn in os.listdir(db_dir) if fn.endswith(".json")]
 
 
-def get_winapi1024_path():
-    return get_this_dir() + os.sep + "apiscout" + os.sep + "data" + os.sep + "winapi1024v1.txt"
-
-
 def get_base_addr(args):
     if args.base_addr:
         return int(args.base_addr, 16) if args.base_addr.startswith("0x") else int(args.base_addr)
@@ -88,8 +84,6 @@ def main():
             db_paths = get_all_db_files()
         for db_path in db_paths:
             scout.loadDbFile(db_path)
-        # load WinApi1024 vector
-        scout.loadWinApi1024(get_winapi1024_path())
         # scout the binary
         results = {}
         if args.import_table_only:

scout.py

@@ -2,23 +2,31 @@
 
 from setuptools import setup, find_packages
 
-
-with open('README.rst') as f:
-    README = f.read()
-
-with open('LICENSE') as f:
-    LICENSE = f.read()
+with open("README.md", "r") as fh:
+    long_description = fh.read()
 
 setup(
     name='apiscout',
-    version='1.0.1',
-    description='Windows API recovery.',
-    long_description=README,
+    version='1.1.0',
+    description='A library for Windows API usage recovery and similarity assessment with focus on memory dumps.',
+    long_description_content_type="text/markdown",
+    long_description=long_description,
     author='Daniel Plohmann',
     author_email='[email protected]',
     url='https://github.com/daniel-plohmann/apiscout',
-    license=LICENSE,
-    # packages=find_packages(exclude=('tests', 'docs')),
-    packages = ["apiscout"],
-    package_data={"apiscout": ["data/winapi1024v1.txt"]},
+    license="BSD 2-Clause",
+    packages=find_packages(exclude=('tests', 'dbs')),
+    package_data={'apiscout': ['data/winapi1024v1.txt', 'data/winapi_contexts.csv', 'data/html_frame.html']},
+    data_files=[
+        ('', ['LICENSE']),
+    ],
+    classifiers=[
+        "Development Status :: 4 - Beta",
+        "License :: OSI Approved :: BSD License",
+        "Operating System :: OS Independent",
+        "Programming Language :: Python :: 2.7",
+        "Programming Language :: Python :: 3",
+        "Topic :: Security",
+        "Topic :: Software Development :: Disassemblers",
+    ],
 )