Add support/example for CPY 8, and non-secondary ESP chips

[tyeth]

I don't know how to change the example to work with circuitpython 8, I think this detracts from the usability hugely. I hoped i could just setup an MQTT client from minimqtt (set broker details) and pass that to adafruit_io with the wifi already connected, but alas no joy without passing socketpool too.
This was my eventual code (no need to connect to wifi in v8 if secrets set in .env):

# Set up the MQTT client
mqtt_client = MQTT(
    broker="io.adafruit.com",
    port=1883,
    username=aio_username,
    password=aio_key,
    socket_pool=socketpool.SocketPool(wifi.radio),
)
print("Connecting to Adafruit IO MQTT broker...")

# Pass the MQTT client to the IO_MQTT constructor
aio_client = IO_MQTT(mqtt_client)
aio_client.connect()
print('Connected to Adafruit IO!')

There are quite a few adafruit devices with ESP32 native chips, so should the ESP32 SPI wifi library still be used, also secrets go elsewhere these days.
Like a settings.toml file with this content:

CIRCUITPY_WIFI_SSID="WIFI_SSID_NAME"
CIRCUITPY_WIFI_PASSWORD="password"
CIRCUITPY_AIO_USERNAME="tyeth"
CIRCUITPY_AIO_KEY="MYSUPERSECRETAPIKEY"

Also on my mind, is the group issue (#97) affecting the data points per minute count of the IO rate limit, or does it not matter (e.g. if I send a single group REST api post with 8 datapoints, compared to sending 8 single REST post calls)?

[tyeth]

if only I'd looked past simple test example

[brentru]

@tyeth Were you able to get this going? We should probably leave this open as the examples should be updated for CPY 8

[tyeth]

yeah, the esp32s2 example is basically it. with os for env secrets, and no need to for wifi connect code if secrets populated.
Think there maybe a bug as was expecting throttle and ban issues on adafruitIO to raise ThrottleError etc, but I had to subscribe to the feeds in the underlying layer of minimqtt and no errors were raised by the library (note: mqtt msgs only triggered on aio_client.loop(0) so assuming that is being called).
Just playing with sen5x on circuitpython, but saved my keys so private repo, excuse possible extraneous code ;)

import time
import board
import busio
from adafruit_io import adafruit_io_errors
from Adafruit_IO import *
from adafruit_io.adafruit_io_errors import AdafruitIO_RequestError, AdafruitIO_ThrottleError, AdafruitIO_MQTTError
from adafruit_io.adafruit_io import IO_MQTT
import os
import wifi
from adafruit_minimqtt.adafruit_minimqtt import MQTT
import socketpool

DATA_POINTS_PER_MINUTE = 60 # set to 30 if not paid adafruit IO plan

# Define global variables to handle rate limits, throttling, and bans
throttle_time = 0
ban_time = 0

# Read the Adafruit IO username and API key from the CircuitPython settings file
aio_username = os.getenv('CIRCUITPY_AIO_USERNAME')
aio_key = os.getenv('CIRCUITPY_AIO_KEY')

# Callback function for 'tyeth/throttle' topic
def on_throttle(client, topic, message):
    global throttle_time
    throttle_time = 15
    try:
        throttle_time = float(message.split(',')[1].split()[0])
        print("Throttle message received: ", message)
    except ValueError as e:
        print("Error parsing throttle message: ", message)
    print("Sleeping for ",throttle_time,"s")
    time.sleep(throttle_time)

# Callback function for 'tyeth/errors' topic
def on_ban(client, topic, message):
    global ban_time
    if(message.find("ban")):
        ban_time = float(message.split()[-2])
        print("Ban (",ban_time,"s) message received: ", message)
        print("Sleeping for ",ban_time,"s")
        time.sleep(ban_time)
        print("Done sleeping due to ban")
    else:
        print("Error message received on topic (",topic,"): ", message)


# pylint: disable=unused-argument
def disconnected(client):
    # Disconnected function will be called when the client disconnects.
    print("Disconnected from Adafruit IO!")


# pylint: disable=unused-argument
def message(client, feed_id, payload):
    # Message function will be called when a subscribed feed has a new value.
    # The feed_id parameter identifies the feed, and the payload parameter has
    # the new value.
    print("Feed {0} received new value: {1}".format(feed_id, payload))

# Set up the MQTT client
mqtt_client = MQTT(
    broker="io.adafruit.com",
    port=1883,
    username=aio_username,
    password=aio_key,
    socket_pool=socketpool.SocketPool(wifi.radio),
)
print("Connecting to Adafruit IO MQTT broker...")

# Pass the MQTT client to the IO_MQTT constructor
aio_client = IO_MQTT(mqtt_client)

while True:
    try:
        aio_client.connect()
        aio_client.subscribe_to_errors()
        aio_client.subscribe_to_throttling()
        aio_client.on_message = message
        aio_client.on_disconnect = disconnected

        mqtt_client.add_topic_callback('tyeth/errors', on_ban)
        mqtt_client.add_topic_callback('tyeth/throttle', on_throttle)

        print('Connected to Adafruit IO!')
        break
    except Exception as e:
        print("Failed to connect, retrying\n", e)
        time.sleep(10)

[tyeth]

Think there maybe a bug as was expecting throttle and ban issues on adafruitIO to raise ThrottleError etc, but I had to subscribe to the feeds in the underlying layer of minimqtt and no errors were raised by the library (note: mqtt msgs only triggered on aio_client.loop(0) so assuming that is being called).

@brentru have you ever seen the throttle or ban errors (the error classes from IO library) be raised? Maybe a bug or maybe a misunderstanding on my part as to how or where to trap them...

[brentru]

I'm sure I've seen them raised at some point, it has been a while since I've worked on this library.

You could try publishing really fast within the while loop to cause IO to temporarily ban/throttle you.

[tyeth]

oh I've been throttle and banned a lot, this is the script I'm running with an scd41 and sen55 but it'll work with scd40/scd42 and sen50/54/55.
It has a button (D2 for reverse TFT) to skip the wait time, so you can hit it fairly quickly (especially with two devices).
You'll need to get my branches of libraries:
https://github.com/tyeth/python-i2c-driver/tree/circuitpython-busio-i2cdevice
https://github.com/tyeth/circuitpython-i2c-sen5x/tree/circuitpython-fix
https://github.com/tyeth/Adafruit_CircuitPython_Logging/tree/patch-1
then use circup install --auto --py
and this is code.py

import supervisor
import time
import board
import busio
from adafruit_io import adafruit_io_errors
from Adafruit_IO import *
from adafruit_io.adafruit_io_errors import AdafruitIO_RequestError, AdafruitIO_ThrottleError, AdafruitIO_MQTTError
from adafruit_io.adafruit_io import IO_MQTT
import os
import storage
import digitalio
import wifi
from adafruit_minimqtt.adafruit_minimqtt import MQTT, MMQTTException
import socketpool

import json
import struct
from adafruit_scd4x import SCD4X
#from sensirion_i2c_scd import Scd4xI2cDevice
from sensirion_i2c_sen5x import Sen5xI2cDevice
from sensirion_i2c_driver import I2cTransceiver,I2cConnection

import adafruit_logging as logging
logger = logging.getLogger(__name__)

DATA_POINTS_PER_MINUTE = 60 # set to 30 if not paid adafruit IO plan
MAX_SENSOR_ERRORS_BEFORE_REBOOT = 10
# NOTE: Display rotation is best fixed in boot.py

# Define global variables to handle rate limits, throttling, and bans
throttle_time = 0
ban_time = 0

# Read the Adafruit IO username and API key from the CircuitPython settings file
aio_username = os.getenv('CIRCUITPY_AIO_USERNAME')
aio_key = os.getenv('CIRCUITPY_AIO_KEY')

#Setup boot toggle pin, also timer skip pin
D2 = digitalio.DigitalInOut(board.D2)



class Sen5xProductType:
    SPS30 = "SPS30"
    SEN50 = "SEN50"
    SEN54 = "SEN54"
    SEN55 = "SEN55"


# Callback function for 'tyeth/throttle' topic
def on_throttle(client, topic, message):
    global throttle_time
    throttle_time = 15
    try:
        throttle_time = float(message.split(',')[1].split()[0])
        print("Throttle message received: ", message)
    except ValueError as e:
        print("Error parsing throttle message: ", message)
    print("Sleeping for ",throttle_time,"s")
    time.sleep(throttle_time)

# Callback function for 'tyeth/errors' topic
def on_ban(client, topic, message):
    global ban_time
    if(message.find("ban")):
        ban_time = float(message.split()[-2])
        print("Ban (",ban_time,"s) message received: ", message)
        print("Sleeping for ",ban_time,"s")
        time.sleep(ban_time)
        print("Done sleeping due to ban")
    else:
        print("Error message received on topic (",topic,"): ", message)


# pylint: disable=unused-argument
def disconnected(client):
    # Disconnected function will be called when the client disconnects.
    print("Disconnected from Adafruit IO!")


# pylint: disable=unused-argument
def message(client, feed_id, payload):
    # Message function will be called when a subscribed feed has a new value.
    # The feed_id parameter identifies the feed, and the payload parameter has
    # the new value.
    print("Feed {0} received new value: {1}".format(feed_id, payload))



D2.direction = digitalio.Direction.INPUT
D2.pull = digitalio.Pull.DOWN
print("D2.value=",D2.value)

# Function to convert MAC address bytes to a string
def bytes_to_mac_string(mac_bytes):
    return ':'.join(['{:02X}'.format(byte) for byte in mac_bytes])

# Get the MAC address as a string
mac_address = bytes_to_mac_string(wifi.radio.mac_address)

def print_wifi_status():
    # elif status == wifi.WiFiState.IDLE:
    #     print("Wi-Fi idle")
    #     print("Connecting to Wi-Fi")
    # elif status == wifi.WiFiState.CONNECTED:
    # elif status == wifi.WiFiState.DISCONNECTED:
    if wifi.radio.connected:
        print("Connected to Wi-Fi")
    else:
        print("Disconnected from Wi-Fi")
        # print("Unknown Wi-Fi status")

print_wifi_status()

# Set up the MQTT client
mqtt_client = MQTT(
    broker="io.adafruit.com",
    port=1883,
    username=aio_username,
    password=aio_key,
    socket_pool=socketpool.SocketPool(wifi.radio),
)
print("Connecting to Adafruit IO MQTT broker...")

# Pass the MQTT client to the IO_MQTT constructor
aio_client = IO_MQTT(mqtt_client)

while True:
    try:
        aio_client.connect()
        aio_client.subscribe_to_errors()
        aio_client.subscribe_to_throttling()
        aio_client.on_message = message
        aio_client.on_disconnect = disconnected

        mqtt_client.add_topic_callback('tyeth/errors', on_ban)
        mqtt_client.add_topic_callback('tyeth/throttle', on_throttle)

        print('Connected to Adafruit IO!')
        break
    except Exception as e:
        print("Failed to connect, retrying\n", e)
        time.sleep(10)


# Set up the I2C buses and the ADS1x15 instances
i2c_buses =  [board.I2C(), board.STEMMA_I2C()] if hasattr(board, "STEMMA_I2C") else [board.I2C()]
print(i2c_buses)
# ads_list = []
# channels = []



# for i, i2c in enumerate(i2c_buses):
#     for addr in range(0x48, 0x4C):
#         try:
#             ads = ADS1115Wrapper(i2c=i2c, address=addr)
#                 # :param ~busio.I2C i2c: The I2C bus the device is connected to.
#                 # :param float gain: The ADC gain.
#                 # :param int data_rate: The data rate for ADC conversion in samples per second.
#                 #                       Default value depends on the device.
#                 # :param Mode mode: The conversion mode, defaults to `Mode.SINGLE`.
#                 # :param int address: The I2C address of the device.        
#             print(ads.address)
#             ads_list.append(ads)
#             channels.append([AnalogIn(ads.ads, ADS.P0),
#                              AnalogIn(ads.ads, ADS.P1),
#                              AnalogIn(ads.ads, ADS.P2),
#                              AnalogIn(ads.ads, ADS.P3)])
#             break
#         except ValueError:
#             continue

# # Define feed names
# boardname="UnknownBoard"
# try:
#     boardname=board.board_id
# except:
#     print("No board id found")

# group_name = boardname + "-" + mac_address.replace(":", "")
# feed_names = []

# for i, ads in enumerate(ads_list):
#     feed_names.extend([group_name.replace("_","-") + f"_i2c{i}_ads1x15_0x{ads.address:X}_volts_A{j}".replace("_","-") for j in range(4)])



# Define feed names
boardname = board.board_id if hasattr(board, "board_id") else "UnknownBoard"
group_name = boardname.replace("_", "-") + "-" + mac_address.replace(":", "")
feed_names = {
    "scd4x": {
        "co2": group_name + "-scd4x-co2",
        "temperature": group_name + "-scd4x-temperature",
        "humidity": group_name + "-scd4x-humidity",
    },
    "sen5x": {
        "ppm-1": group_name + "-sen5x-ppm-1",
        "ppm-2.5": group_name + "-sen5x-ppm-2.5",
        "ppm-4": group_name + "-sen5x-ppm-4",
        "ppm-10": group_name + "-sen5x-ppm-10",
        "temperature": group_name + "-sen5x-temperature",
        "humidity": group_name + "-sen5x-humidity",
        "voc": group_name + "-sen5x-voc",
        "nox": group_name + "-sen5x-nox",
    },
}

# Discover SCD4x and SEN5x devices on I2C buses
scd4x_device = None
sen5x_device = None

SCD4X_DEFAULT_ADDRESS = 0x62
SEN5X_DEFAULT_ADDRESS = 0x69

for i2c in i2c_buses:
    try:
        scd4x_device = SCD4X(i2c, SCD4X_DEFAULT_ADDRESS)
        print(f"SCD4x device found on I2C bus {i2c}, #{scd4x_device.serial_number}")
        scd4x_device.start_periodic_measurement()
        break
    except (OSError,ValueError):
        print("SCD4x sensor not detected")
        continue

sen5x_product = "SEN5x"
for i2c in i2c_buses:
    try:
        
        transceiver = I2cTransceiver(i2c, SEN5X_DEFAULT_ADDRESS)
        sen5x_device = Sen5xI2cDevice(I2cConnection(transceiver))
        sen5x_device.device_reset()
        print("Resetting SEN5x sensor, waiting 1.1 seconds...")
        time.sleep(1.1)
        # sen5x_device = Sen5xI2cDevice(i2c, SEN5X_DEFAULT_ADDRESS)
        sen5x_product = sen5x_device.get_product_name()
        print(f"SEN5x device found on I2C bus {i2c}, product type: {sen5x_product}, #{sen5x_device.get_serial_number()}")
        sen5x_device.start_measurement()
        break
    except (OSError,ValueError):
        print("SEN5x sensor not detected")
        continue


# # Loop forever, reading the ADC and publishing data to Adafruit IO
# while True:
#     channel_total=0
#     for i, chans in enumerate(channels):
#         for j, chan in enumerate(chans):
#             if throttle_time > 0:
#                 print(f"Throttling data for {throttle_time} seconds")
#                 for k in range(throttle_time):
#                     time.sleep(0.5)
#                     if(D2.value==True):
#                         print("Throttle cancelled")
#                         break
#                     time.sleep(0.5)
#                 throttle_time = 0
            
#             # Read the ADC value and convert it to a voltage reading
#             val = chan.value
#             volt = val * 3.3 / 65535.0
#             print("Attempting to publish value (",volt,",",val,") to feed ",feed_names[i*4+j])
#             # Publish the voltage reading to Adafruit IO
#             # try:
#             aio_client.publish(feed_names[i * 4 + j], volt)
#             aio_client.loop(0)
#             print(".")
#             # except AdafruitIO_ThrottleError as e:
#             #     print("Data throttled, msg: ",e)
#             #     time.sleep(10)
#             # except AdafruitIO_MQTTError as e:
#             #     print("MQTT ERROR, msg: ",e)
#             #     time.sleep(10)
#             # except AdafruitIO_RequestError as e:
#             #     print("Request Error, msg: ",e)
#             #     time.sleep(10)
#             # except Exception as e:
#             #     print("Failed to publish to feed",e)
#             #     #print stacktrace from exception e.with_traceback()
#             #     print("Traceback: ",)
                
#             #     time.sleep(10)

#             channel_total+=1

#     # Wait for a short time before reading the ADC again
#     time.sleep(0.5)
#     # wait for datapoints total to not flood rate limit of 30 per minute
#     datapoints_per_minute = 1 if channel_total>DATA_POINTS_PER_MINUTE else DATA_POINTS_PER_MINUTE / channel_total
#     print('Sleeping for ',(60/datapoints_per_minute),'seconds for',channel_total,'datapoints')
#     sleep_time = 60/datapoints_per_minute
#     for k in range(int(sleep_time)):
#         time.sleep(1)
#         if(D2.value==True):
#             print("Sleep cancelled")
#             break


scd4x_errors = 0
sen5x_errors = 0

# Loop forever, reading the sensor data and publishing it to Adafruit IO
while True:
    try:
        if scd4x_device:
            try:
                if scd4x_device.data_ready:
                    print(f"Publishing SCD4X data: CO2: {scd4x_device.CO2}, Temperature: {scd4x_device.temperature}, Humidity: {scd4x_device.relative_humidity}")
                    aio_client.publish(feed_names["scd4x"]["co2"], scd4x_device.CO2)
                    aio_client.publish(feed_names["scd4x"]["temperature"], scd4x_device.temperature)
                    aio_client.publish(feed_names["scd4x"]["humidity"], scd4x_device.relative_humidity)
                    scd4x_errors = 0
                    aio_client.loop(0)
            except OSError as e:
                print("SCD4x Error: ",e)
                scd4x_errors+=1

        if sen5x_device:
            try:
                if sen5x_device.read_data_ready():
                    sen5x_data = sen5x_device.read_measured_values()
                    print(f"Publishing {sen5x_product} data: PM1.0: {sen5x_data.mass_concentration_1p0.physical}, PM2.5: {sen5x_data.mass_concentration_2p5.physical}, PM4.0: {sen5x_data.mass_concentration_4p0.physical}, PM10: {sen5x_data.mass_concentration_10p0.physical}")
                    aio_client.publish(feed_names["sen5x"]["ppm-1"], sen5x_data.mass_concentration_1p0.physical)
                    aio_client.publish(feed_names["sen5x"]["ppm-2.5"], sen5x_data.mass_concentration_2p5.physical)
                    aio_client.publish(feed_names["sen5x"]["ppm-4"], sen5x_data.mass_concentration_4p0.physical)
                    aio_client.publish(feed_names["sen5x"]["ppm-10"], sen5x_data.mass_concentration_10p0.physical)
                    aio_client.loop(0)

                    if sen5x_product in (Sen5xProductType.SPS30, Sen5xProductType.SEN54, Sen5xProductType.SEN55):
                        print(f"Publishing {sen5x_product} data: VOC: {sen5x_data.voc_index.scaled} Temperature: {sen5x_data.ambient_temperature.degrees_celsius} Humidity: {sen5x_data.ambient_humidity.percent_rh}")
                        aio_client.publish(feed_names["sen5x"]["voc"], sen5x_data.voc_index.scaled)
                        aio_client.publish(feed_names["sen5x"]["temperature"], sen5x_data.ambient_temperature.degrees_celsius)
                        aio_client.publish(feed_names["sen5x"]["humidity"], sen5x_data.ambient_humidity.percent_rh)
                        aio_client.loop(0)

                    if sen5x_product == Sen5xProductType.SEN55:
                        print(f"Publishing {sen5x_product} data: NOx: {sen5x_data.nox_index.scaled}")
                        aio_client.publish(feed_names["sen5x"]["nox"], sen5x_data.nox_index.scaled)
                        aio_client.loop(0)
            except OSError as e:
                print("SEN5x Error: ",e)
                sen5x_errors+=1
    except MMQTTException as e:
        print("MQTT Error: ",e)
        print("Rebooting...")
        supervisor.reload()

    # If we've had 5 errors in a row, reboot
    if scd4x_errors >= MAX_SENSOR_ERRORS_BEFORE_REBOOT or sen5x_errors >= MAX_SENSOR_ERRORS_BEFORE_REBOOT:
        print("Too many sensor errors, rebooting...")
        supervisor.reload()

    # Wait for a short time before reading the sensor data again
    time.sleep(0.5)
    # wait for datapoints total to not flood rate limit of 30 per minute
    channel_total = 3 + 4  # Number of data points (3 for SCD4x, 4 for SEN5x)
    if sen5x_product in (Sen5xProductType.SPS30, Sen5xProductType.SEN54, Sen5xProductType.SEN55):
        channel_total += 3  # Add 1 for VOC data point
    if sen5x_product == Sen5xProductType.SEN55:
        channel_total += 1  # Add 1 for NOx data point

    datapoints_per_minute = 1 if channel_total > DATA_POINTS_PER_MINUTE else DATA_POINTS_PER_MINUTE / channel_total
    print('Sleeping for ', (60 / datapoints_per_minute), 'seconds for', channel_total, 'datapoints')
    sleep_time = 60 / datapoints_per_minute
    for k in range(int(sleep_time)):
        time.sleep(1)
        if D2.value == True:
            print("Sleep cancelled")
            break

[tyeth]

@brentru it's not intelligent in the fact it double waits for throttle messages (if a new one comes while throttling), but I consider that a "feature" rather than bug ;)

Also upon banning I was seeing a disconnected error, but now seem to first see a ping response failure (mMQTTexception). Wonder if something changed or if I started subscribing to more things (most likely as I added on_disconnected and on_message subsbriptions)