External connectors not working behind a proxy - example cisa-known-exploited-vulnerabilities

[martinlindstrand]

Description

We are working behind proxy. We are running from docker compose setup.

Environment

1. OS Red hat 8.9
2. OpenCTI version: 6.0.10
3. Other environment details:

Behind a proxy, using default setup from https://docs.opencti.io/latest/deployment/installation/ except adding adding external connectors. cisa-known-exploited-vulnerabilities in the docker-compose.yml and adding support for proxy.
http_proxy=http://proxy.com < not the real proxy adress.
https_proxy=http://proxy.com
no_proxy = internalipadresses....

Steps to create the smallest reproducible scenario:
Adding the external connectors cisa-known-exploited-vulnerabilities.

Expected Output

It should import the information from cisa

Actual Output

from Cisa:
{"timestamp": "2024-04-27T04:32:51.027442Z", "level": "ERROR", "name": "CISA Known Exploited Vulnerabilities", "message": "the JSON object must be str, bytes or bytearray, not NoneType", "exc_info": "Traceback (most recent call last):\n File "/opt/opencti-connector-cisa-known-exploited-vulnerabilities/main.py", line 284, in process_data\n cisa_data = json.loads(cisa_data)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/json/init.py", line 339, in loads\n raise TypeError(f'the JSON object must be str, bytes or bytearray, '\nTypeError: the JSON object must be str, bytes or bytearray, not NoneType"}
{"timestamp": "2024-04-27T04:35:59.434021Z", "level": "ERROR", "name": "CISA Known Exploited Vulnerabilities", "message": "Error retrieving url https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json: <urlopen error [Errno 110] Operation timed out>", "exc_info": "Traceback (most recent call last):\n File "/usr/local/lib/python3.11/urllib/request.py", line 1348, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File "/usr/local/lib/python3.11/http/client.py", line 1303, in request\n

same issue on urlhous.

Additional information

If you adding in the main.py in cisa-known-exploited-vulnerabilities on line 80 below try:

     proxy_url = os.environ.get('http_proxy')
     proxies = {"http": proxy_url, "https": proxy_url}
     if proxy_url:
        response = requests.get(url, proxies=proxies)
        return response.text

don't forget to add import requests

It will start working. it will be using the proxy settings and fetch the information. I suggest this type of change should be check on all the external connectors to make it work with proxy.

Other external connectors like alienvault are working fine behind the proxy.

Screenshots (optional)

[helene-nguyen]

Hi @martinlindstrand,
Thank you for your workaround. Maybe I can add another suggestion.
Because environment variables are case sensitive in Python scripts, when you add them to your `docker-compose.yml' file, you need to make sure that any environment variable you add is uppercase.

For proxy, you need to add it as the following in you docker-compose.yml file:

- HTTP_PROXY=...
- HTTPS_PROXY=...
- NO_PROXY=...

Could you try by setting this in uppercase and give us an update?

[martinlindstrand]

That did not work.

For reference when using
HTTP_PROXY
HTTPS_PROXY
NO_PROXY

The response for using this code:
proxy_url = os.environ.get('http_proxy')
proxies = {"http": proxy_url, "https": proxy_url}
print(proxy_url)

RESPONSE:
python main.py
None <------ response from the print(proxy_url)
python3 main.py
None <------ response from the print(proxy_url)

When using with lowercase
http_proxy
https_proxy
no_proxy

python main.py
http://proxy.com <------ response from the print(proxy_url)

[helene-nguyen]

@martinlindstrand Following our discussion on Slack, I close this issue but it can be re-opened if needed :)

[CharlesMagne2001]

Hello,
I have the same problem. I am working under proxy.
I tried to use the connectors : ipinfo and shodan.
I here is my yml part:

opencti:
image: opencti/plateform:6.0.5
...
...
connector-shodan:
image: opencti/connector-shodan:6.1.10
environment:
- https_proxy=http://xxx.xxx.xxx.xxx:xxxxx
- HTTPS_PROXY=http://xxx.xxx.xxx.xxx:xxxxx
- http_proxy=http://xxx.xxx.xxx.xxx:xxxxx
- HTTP_PROXY=http://xxx.xxx.xxx.xxx:xxxxx
- no_proxy=xxx.xxx.xxx.xxx,opencti
- NO_PROXY=xxx.xxx.xxx.xxx,opencti
- OPENCTI_URL=http://opencti:8080
- OPENCTI_TOKEN=xxxxxx-xxx-xxxx-xxx-xxxx
- CONNECTOR_ID=xxxxxxxxxxxxxxxxxxxx
- CONNECTOR_NAME=Shodan
- CONNECTOR_SCOPE=IPv4-Addr
- CONNECTOR_AUTO=true
- CONNECTOR_LOG_LEVEL=error
- SHODAN_TOKEN=yyyyyyyyyy
- SHODAN_MAX_TLP=TLP:AMBER
- SHODAN_CREATE_INDICATORS=true
- SHODAN_DEFAULT_SCORE=50
restart: always
depends_on:
- opencti
connector-ipinfo:
image: opencti/connector-ipinfo:6.1.10
environment:
- https_proxy=http://xxx.xxx.xxx.xxx:xxxxx
- HTTPS_PROXY=http://xxx.xxx.xxx.xxx:xxxxx
- http_proxy=http://xxx.xxx.xxx.xxx:xxxxx
- HTTP_PROXY=http://xxx.xxx.xxx.xxx:xxxxx
- no_proxy=xxx.xxx.xxx.xxx,opencti
- NO_PROXY=xxx.xxx.xxx.xxx,opencti
- OPENCTI_URL=http://opencti:8080
- OPENCTI_TOKEN=xxxxxx-xxx-xxxx-xxx-xxxx
- CONNECTOR_ID=xxxxxxxxxxxxxxxxxxxx
- CONNECTOR_NAME=IpInfo
- CONNECTOR_SCOPE=IPv4-Addr
- CONNECTOR_AUTO=true
- CONNECTOR_CONFIDENCE_LEVEL=75 # From 0 (Unknown) to 100 (Fully trusted)
- CONNECTOR_LOG_LEVEL=error
- IPINFO_TOKEN=yyyyy
- IPINFO_MAX_TLP=TLP:AMBER
restart: always
depends_on:
- opencti

and I get the same error below when I enrich a IPv4 from shodan or ipinfo :
SHODAN :
{"timestamp": "2024-06-11T11:23:17.739011Z", "level": "ERROR", "name": "Shodan", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 268, in _data_handler\n opencti_entity = do_read(id=entity_id, withFiles=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py", line 750, in read\n result = self.opencti.query(query, {"id": id})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 363, in query\n raise ValueError(value_error)\nValueError: {'name': 'Unknown type "TrackingNumber".', 'error_message': 'Unknown type "TrackingNumber".'}"}

IPINFO:
{"timestamp": "2024-06-11T11:22:31.125056Z", "level": "ERROR", "name": "IpInfo", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 268, in _data_handler\n opencti_entity = do_read(id=entity_id, withFiles=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py", line 750, in read\n result = self.opencti.query(query, {"id": id})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 363, in query\n raise ValueError(value_error)\nValueError: {'name': 'Unknown type "TrackingNumber".', 'error_message': 'Unknown type "TrackingNumber".'}"}
{"timestamp": "2024-06-11T11:23:17.758439Z", "level": "ERROR", "name": "IpInfo", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n File "/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py", line 268, in _data_handler\n opencti_entity = do_read(id=entity_id, withFiles=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py", line 750, in read\n result = self.opencti.query(query, {"id": id})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 363, in query\n raise ValueError(value_error)\nValueError: {'name': 'Unknown type "TrackingNumber".', 'error_message': 'Unknown type "TrackingNumber".'}"

Could you tell me if the issues are linked to proxies? How to fix them?