[project] pi-top[4] Sensor data Multithreading and Queue

So i am looking into multi threading for a purpose, The Raspberry pi Pico, so am not starting on micropython as its rather limited and little info is about for the Pico right now so yeah, pi-top[4] it is and also this helps others out

Idea
So the idea is to have 1 thread to get data from a sensor and another thread to print the data to the console for step 1 and did not want it to be bloated.

Solution
The solution to this is, obviously use multi threading but also add a queue system
Thread 1: Get sensor data and pass it to the queue
Thread 2: wait for data to be added to the queue and print the data to the console

I have never used threading or queue before so a lot of this was trial and error, starting with a counter and printing the count though threading and queue. and I worked it all out.

Result
Thread 1: gets GPS data (GPGGA, GPGSA, GPRMC, GPZDA) and add them to the queue
Thread 2: waits till the queue is not empty and prints each queue item to the console (GPS data) if the queue is empty then print “waiting for work”, and had to add a sleep timer in there to slow down the thread a little

What i have done here is add some logging to see what is going on and here is the code that was run

import queue, threading, io, serial, pynmea2
from time import sleep
import logging

# Serial setup
gpsSerial = serial.Serial('/dev/ttyS0', 9600, timeout=1.)
gpsIO = io.TextIOWrapper(io.BufferedRWPair(gpsSerial, gpsSerial))

# Enable only NMEA sentences required
gpsSerial.write(b'$PMTK314,0,1,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,1*34\r\n')

nmea_list = ['GGA', 'GSA', 'RMC', 'ZDA']
q = queue.Queue()

def SensorThread(q,name):
    
    while True:
        logging.debug(f'{name} Thread: Started')
        try:
            logging.info(f'{name} Thread: Reading GPS line Data')
            line = gpsIO.readline()
            logging.info(f'{name} Thread: Parsing GPS line Data')
            msg = pynmea2.parse(line)
            logging.info(f'{name} Thread: Checking if NMEA Sentence is wanted')
            if msg.sentence_type in nmea_list:
                logging.info(f'{name} Thread: Adding NMEA Sentence to work queue')
                q.put(msg)
        except serial.SerialException as e:
            print('Device error: {}'.format(e))
            break
        except (pynmea2.ParseError, AttributeError) as e:
            continue


def PrintThread(q,name):
    logging.info(f'{name} Thread: Started')
    while True:
        logging.info(f'{name} Thread: checking to see if the work queue is empty')
        if not q.empty():
            logging.info(f'{name} Thread: printing work')
            print(f'Print Thread: {q.get()}')
        
        sleep(0.1)


if __name__ == "__main__":
    format = "%(asctime)s: %(message)s"
    logging.basicConfig(format=format, level=logging.INFO,
                        datefmt="%H:%M:%S")

    logging.info(f'Creating Sensor Thread')
    qt = threading.Thread(target=SensorThread, args=(q,'Sensor'))
    logging.info(f'Creating Print Thread')
    pt = threading.Thread(target=PrintThread, args = (q,'Print'))
    logging.info(f'Starting Sensor Thread')
    qt.start()
    logging.info(f'Starting Sensor Thread')
    pt.start()

More development for this will continue on this till get it to display GPS information on the display, may even add functionality to use the mini screen buttons to scroll though different data

1 Like

Well an update already :slight_smile:

I have the multithead working as you know, but i have it also displaying on the screen. Thanks to PIL i can save what is displayed on the screen, here ya go

pi-top_gps_display

the 1st second is the script starting, only added the saving of the display because i have found another issue with my pi-top[4]. will look into that later.

Anyways, back on topic. The code is currently uncommented but is on my pi-top[4] GPS repo here:-
File name: pitop-gps-serial-multithread.py

https://github.com/CAProjects/pitop4-gps-serial

@duwudi didnt think i would have it complete so soon :slight_smile:

3 Likes

I was going to comment that there is a “_thread” library for micropython… However, it is currently somewhat limited in its capabilities. For the best multi-threaded support, we are going to have to wait for FreeRTOS to get a port to the RP2040, but even then, it is a C/C++ based system.

It would be awesome to see FreeRTOS get a python port. Or a whole new RTOS designed around python would be pretty awesome too!

I have seen a video of the Pico using _thread and it worked rather well, now that I have this done i can do a port of it for the Pico and will let you know how it runs, i have a display for the Pico ordered so will have a better way to see an output rather than a debug console. down side to a new platform is waiting for ports to become available.

At the moment from what i have read the following are getting ported

  • FreeRTOS
  • RUST
  • RT-Thread OS
  • Ardiono

Yeah, I’m excited to see what this brings. I’m just waiting for Q2 when the RP2040 becomes available to source. Wanting to make a dev board with the NINA module and a crypto-chip to create a pico iot style kit. :slight_smile:

at the moment with the GPS work i have been doing, I am going to focus the Pico stuff on a GPS data logger, which next I will be looking at getting a micro SD card breakout and have a look at writing data to that too and also be able to start and stop logging from push buttons.

1 Like

@CAProjects: Thank you for your excellent work. Have tested the new code with multithreading and it works perfectly.

I tweeked the code a little, I think line 24 should read ‘fix’ : “” instead of True otherwise it will display fix is Yes eventhoug there is no fix.

Screenshot_20210226-173403_Gallery

it should be set to ‘No’ but does not really matter either way as there is no checking to se if there is a fix or not with this, was my oversight on that. nice to see you got it working. for the fix, i use a 1 line if statement that does not check for fix mode, just checks the fix type (the 2D/3D type) because pynmea2 is a little odd with GPS fix, would have been nice for it to expose true or false for fix, instead they dont have a variable for that

the multi threaded way of doing things make the processing of the data a lot faster and i made the dictionary keys so that if there is no GPS data it will just be blank anyways. Like all my code i rewrite them many times to get better optimization and add more features.

The speed calculation is hard coded to MPH and will add comments to my code soon and give the calc values for different units. I also want to handle Lat and Lon better. pynmea2 is a little strange as it mainly calculates in DD instead of proper lat and lon coords so i had to add code to check if degrees was + or - to display N or S and W or E. so its all a little messy at the moment and think i will go back to manual method as i think i have cleaner code with it instead of using a 3rd party module.

I may even make a module anyways at some point for NMEA and make everything easily available instead of some things but I don’t know yet on that one. 1 little project can spark others >.< and get you off track on what your actual goal is hehe

@Luis just a question, did you have to add core_freq =250 to your /boot/config.txt to get your GPS working. maybe just a UART thing, i get some strange happenings with the screen setting it, as it has to be set to be able to read via UART. you will see what i mean later on the Sessions call what i mean by that

Lastly, i really want to flesh out a GPS module for the pi-top[4] and have it integrated to the menusystem so you can go in to settings and enable logging mode etc. now that would be a really nice project to do for the pi-top

1 Like

Thanks for all the details, really appreciate the time and effort for explaining. I’ve learned a lot of new things from you with this project. I think the manual way might cause issues with multi constellation gps receivers since you have to tinker around with nmea sentences, in the new code nmea_list = [‘GGA’, ‘GSA’, ‘RMC’, ‘ZDA’] is better because it doesnt seem to matter if it is GPGGA or GNGGA found a document somewhere that showed that these had the same structure. But having it specified in the code as GPGGA at least from what I noticed threw key errors for latitude and longitude.

With regards to core_freq = 250, I don’t have that added to config.txt. GPS and miniscreen work fine without it in my case. But I am using Raspbian OS (Buster). Hope this helps.

1 Like

GN sentences are GNSS/GLONASS/GALILEO systems and need to have hardware that supports it

GP sentences are able to be received by all GPS receivers (Its industry standard) so is why i focus on GP sentences and not GN, mainly as the Adafruit Ultimate GPS is not GLONASS/GALILEO compatible/enabled and cannot be.

Its something I will be looking at in the future but for now GP sentences is the more robust and generic for the average user at this time.

this is because pynmea2 parses the sentences and i use those to check for the sentences i wanted to get data from, i don’t know if it auto detects GLONASS/GALILEO sentences or not as i have not looked into the module that much