"invalid character '<' looking for beginning of value"

[ohshitgorillas]

Hello,

First I want to say this is a very cool and interesting app, I love the statistics output feature. Recently, however, my deployment started giving me the following errors:

W0329 14:54:20.989396       1 metrics.go:128] Failed to obatin the geohash of 114.204.218.154: invalid character '<' looking for beginning of value.
I0329 14:54:33.933339       1 client.go:99] CLOSE host=114.204.218.154 port=36798 time=13.00694231 bytes=170
I0329 14:55:00.133239       1 client.go:58] ACCEPT host=152.32.210.193 port=32194 n=2/4096
W0329 14:55:00.205932       1 metrics.go:128] Failed to obatin the geohash of 152.32.210.193: invalid character '<' looking for beginning of value.
I0329 14:55:13.139388       1 client.go:99] CLOSE host=152.32.210.193 port=32194 time=13.006089047 bytes=252
I0329 14:55:48.516476       1 client.go:58] ACCEPT host=103.103.0.216 port=54992 n=2/4096
W0329 14:55:48.582872       1 metrics.go:128] Failed to obatin the geohash of 103.103.0.216: invalid character '<' looking for beginning of value.
I0329 14:56:01.521648       1 client.go:99] CLOSE host=103.103.0.216 port=54992 time=13.005138886 bytes=164

This came seemingly out of nowhere as the pod was working just fine when I went to bed last night, but I woke up to these errors. There isn't an error before the pod switches from successfully geohashing to these errors.

Below is my deployment.yaml file for K8s:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: endlessh
  annotations:
    keel.sh/policy: all
  labels:
    app: endlessh
spec:
  replicas: 1
  selector:
    matchLabels:
      app: endlessh
  template:
    metadata:
      labels:
        app: endlessh
    spec:
      hostNetwork: true
      nodeSelector:
        kubernetes.io/hostname: obsidiana
      containers:
        - name: endlessh  
          image: shizunge/endlessh-go
          args: 
            - "-logtostderr"
            - "-v=1"
            - "-enable_prometheus"
            - "-port=22420"
            - "-geoip_supplier=ip-api"

[shizunge]

Workaround: use max-mind-db and Geo(Lite)2 City database.

I see, you are using ip-api

[ohshitgorillas]

I tried switching the argument -geoip_supplier=max-mind-db but now the metrics page is blank... edit: metrics page is back

Does this require the -max_mind_db argument to be set? Can you please provide an example of using the -max_mind_db argument, or how to use the Geo(Lite)2 City database?

[shizunge]

1. open a maxmind account https://www.maxmind.com
2. Create a new license key. There are license for free database
3. Download a database, you may use https://github.com/maxmind/geoipupdate
4. Mount the database to endlessh-go container.
5. set -max_mind_db to the path of the database.

https://github.com/shizunge/endlessh-go/tree/main/examples/docker-maxmind

[ohshitgorillas]

That works, thank you!

A friend of mine suggested that the error using ip-api might be that I was sending too many queries and now my IP is blacklisted, so I'm getting an HTML error response instead of the usual response, hence why the response unexpectedly contains '<'. Perhaps I can set up a temporary container to test this, if I know what the command being passed should be?

[shizunge]

A friend of mine suggested that the error using ip-api might be that I was sending too many queries and now my IP is blacklisted, so I'm getting an HTML error response instead of the usual response, hence why the response unexpectedly contains '<'. Perhaps I can set up a temporary container to test this, if I know what the command being passed should be?

I am not I understand your request. You previous setup seems being able to trigger the issue.

The code to handle ip-api response is at geohashAndLocationFromIpapi