TT-12074 Check and remove empty values from resulting set

[monrax]

Description

Empty values from unsuccessful pipelined transactions in GetAndDeleteSet end up in the result slice as empty interfaces. This can cause issues downstream as outlined in the related issue

Related Issue

Issue #768

Motivation and Context

Errors that may occur during data retrieval from Redis (such as i/o errors), should not propagate downstream. Empty values wrapped as interfaces cause other errors downstream, such as type assertion errors. By checking for empty values, no empty interfaces should be returned for decoding during later stages of the purging process.

How This Has Been Tested

• Clone Tyk Pro demo repo
• Add both Resurface and HTTP Bin services to the docker-compose.yml file:

version: '3.9'
services:
  tyk-dashboard:
    image: tykio/tyk-dashboard:v5.1
    container_name: tyk-dashboard
    environment:
      - TYK_DB_LICENSEKEY=${TYK_DB_LICENSEKEY}
      - TYK_DB_STORAGE_MAIN_TYPE=postgres
      - TYK_DB_STORAGE_MAIN_CONNECTIONSTRING=user=postgres password=topsecretpassword host=tyk-postgres port=5432 database=tyk_analytics
    depends_on:
      tyk-postgres:
        condition: service_healthy
    ports:
      - "3000:3000"
    env_file:
      - ./confs/tyk_analytics.env
    networks:
      - tyk

  tyk-gateway:
    image: tykio/tyk-gateway:v5.1
    container_name: tyk-gateway
    ports:
      - "8080:8080"
    env_file:
      - ./confs/tyk.env
    networks:
      - tyk

  tyk-pump:
    image: tykio/tyk-pump-docker-pub:v1.8
    container_name: tyk-pump
    env_file:
      - ./confs/pump.env
    environment:
        - TYK_PMP_PUMPS_RESURFACEIO_TYPE=resurfaceio
        - TYK_PMP_PUMPS_RESURFACEIO_META_URL=http://tyk-resurface:7701/message
        - TYK_PMP_PUMPS_RESURFACEIO_META_RULES=include debug
        - TYK_PMP_DONTPURGEUPTIMEDATA=true
    depends_on:
      tyk-postgres:
        condition: service_healthy
    networks:
      - tyk

  tyk-redis:
    image: redis
    container_name: tyk-redis
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    networks:
      - tyk

  tyk-postgres:
    image: postgres:latest
    container_name: tyk-postgres

    environment:
      - POSTGRES_DB=tyk_analytics
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=topsecretpassword

    ports:
      - "5432:5432"

    volumes:
      - postgres-data:/var/lib/postgresql/data

    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5

    networks:
      - tyk

  tyk-resurface:
    image: resurfaceio/resurface:3.5.36
    container_name: tyk-resurface
    ports:
      - "7700:7700"
      - "7701:7701"
    volumes:
      - resurface-data:/db
    networks:
      - tyk
  
  httpbin:
    image: kennethreitz/httpbin
    container_name: tyk-httpbin
    ports:
      - "8000:80"
    networks:
      - tyk

  httpbin2:
    image: kennethreitz/httpbin
    container_name: tyk-httpbin-2
    ports:
      - "8001:80"
    networks:
      - tyk

volumes:
  redis-data:
  postgres-data:
  resurface-data:

networks:
  tyk:

• Modify the Resurface pump backend is configured without any timeout.

• Import the following HTTP Bin Tyk API definition:

{
    "name": "httpbin",
    "slug": "httpbin",
    "api_id": "httpbin-api",
    "org_id": "test-org",
    "use_keyless": true,
    "version_data": {
        "not_versioned": true,
        "versions": {
            "Default": {
                "name": "Default",
                "expires": ""
            }
        }
    },
    "proxy": {
        "target_url": "",
        "target_list": [
            "http://httpbin",
            "http://httpbin2"
        ],
        "strip_listen_path": true,
        "enable_load_balancing": true,
        "listen_path": "/http-bin/",
        "disable_strip_slash": true
    },
    "active": true,
    "enable_detailed_recording": true
}

• Use a load testing application like bombardier to send a several concurrent requests to the /image/png endpoint aiming for >4000 RPS (concurrency level of 200 with 500k requests)
• Wait for a few minutes. If using the current Tyk Pump release: the number of purged records will start to build up to a point where the pump can't keep up with the new records being generated. Keep an eye on the Tyk Pump logs. After the first i/o error log occurs, many EOF error logs will follow.

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• Make sure you are requesting to pull a topic/feature/bugfix branch (right side). If pulling from your own
  fork, don't request your master!
• Make sure you are making a pull request against the master branch (left side). Also, you should start
  your branch off our latest master.
• My change requires a change to the documentation.
  • If you've changed APIs, describe what needs to be updated in the documentation.
• I have updated the documentation accordingly.
• Modules and vendor dependencies have been updated; run go mod tidy && go mod vendor
• I have added tests to cover my changes.
• All new and existing tests passed.
• Check your code additions will not fail linting checks:
  • go fmt -s
  • go vet

[sweep-ai]

Apply Sweep Rules to your PR?

• Apply: All new business logic should have corresponding unit tests.
• Apply: Refactor large functions to be more modular.
• Apply: Add docstrings to all functions and file headers.

This is an automated message generated by Sweep AI.