Page Links
Raspberry Pi Example: Serial Data Logger with Display

Overview

This example project describes how to create a serial data logger using a Raspberry Pi. The example provides a Python script that is automatically started at power-on to receive serial data from an external source using the 'serial0' UART input, and then timestamps and logs the data to an HDMI display and a connected USB drive. This example uses an RPi mounted to the back of a SunFounder 10" LCD display, however any HDMI display can be used. This example uses an MSP430 development board to stream serial data, however alternate serial data sources can also be used.


Hardware Used

This example used a Raspberry Pi 3 Model B mounted to the back of a SunFounder 10" LCD display, as shown below. The example setup also included a 5V 2.5A power supply, and a standard wireless keyboard and mouse combo. In order to log data a USB drive is also needed. I used a SanDisk Cruzer Fit 8GB drive. Any USB drive should work, but the small form factor of this drive hides nicely behind the display and provides a clean looking setup.

sunfounder_rear2_trans_300x319 image SunFounder 10" LCD Display with Raspberry Pi 3 Model B
+

SanDiskCruzerFit image
SanDisk Cruzer Fit USB Drive
or similar

IMPORTANT: Make sure the Raspberry Pi has been made "Project Ready" before proceeding.

IMPORTANT: Make sure the USB Drive has been formatted as FAT (e.g. FAT32), and that the drive is given the label "USBDRIVE", and that a textfile named "validate.txt" is created at the top level of the drive. The reason why this is necessary will be explained later.


Required Connections

The example project uses serial data coming into the UART RX (Serial Input) pin 10. When the Raspberry Pi is mounted on the back of the LCD display, this is the 5th pin from the left on the top row. It is marked by a yellow dot on the image below. This pin must be connected to the serial output of the device that will provide the data to be logged. A ground reference is also required. That connection was made using the pin 6 ground. This is the 3rd pin from the left on the top row, and is marked with a green dot in the image below.

pi2-serial-connections2_600x125.png image

In order to provide a stream of serial data for the example, the MSP430 4-bit counting example was modified to include the MSP430F5529LP_UART library, and to assemble and transmit a serial message in the main loop each time the counter was incremented. The code added to the example looked like this:


sprintf(string, "%3d, %3X, %d%d%d%d%d%d%d%d  \r\n", s_count_u8, s_count_u8, BYTETOBINARY(s_count_u8));
SendSerialMsg(string, strlen(string));

The format of the data is not important, although comma-separated data makes it easy to import into applications such as Excel. The important part is that the string is terminated by a '\n' new line character. This informs the Python script running on the Raspberry Pi that the data stream is complete which triggers the logging of the bytes received. The '\r' carriage return character is ignored by the script and is not captured or logged. It is included to make the data stream compatible with both Windows and Linux.



Example Code

The datalogger.py Python script looks like this:

#!/usr/bin/env python
"""
/* ######################################################################### */
/*
 * This file was created by www.DavesMotleyProjects.com
 *
 * This software is provided under the following conditions:
 * Permission is hereby granted, free of charge, to any person obtaining
 * a copy of this software and associated documentation files (the
 * 'Software'), to deal in the Software without restriction, including
 * without limitation the rights to use, copy, modify, merge, publish,
 * distribute, sublicense, and/or sell copies of the Software, and to
 * permit persons to whom the Software is furnished to do so, subject to
 * the following conditions:
 *
 * THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
 * IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
 * CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
 * TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
 * SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 *                                                                           */
/* ######################################################################### */
"""

import io
import os
import time
import datetime
import serial

from time import sleep      # to create delays
from locale import format   # to format numbers
from shutil import rmtree   # to remove directories


PathValid = False
FileName = ""
FilePath = ""
dataIndex = 0

FAIL_PATH = "/media/pi/USBDRIVE1/"
DRIVE_PATH = "/media/pi/USBDRIVE/"
AUTH_PATH = DRIVE_PATH + "validate.txt"

MAX_ROWS_IN_FILE = 65535


"""############################################################################

    Function:       chk_usb_on_start
    Description:    This function is called at the start of the program
    execution and provides a helpful message to the user on the approprate
    actions to take to start datalogging, if a valid usb drive was not found.

############################################################################"""


def chk_usb_on_start():

    global PathValid, DRIVE_PATH, AUTH_PATH


    # if the "validate.txt" file is not found on startup, prompt the user to
    # install the appropriate usb drive
    if not (os.path.exists(AUTH_PATH)):
        print "\n"
        print "To start datalogging, insert a usb drive that is formatted as FAT, with the "
        print "label 'USBDRIVE', that has a text file named 'validate.txt' at the top level "
        print "of the usb drive.\n"

        while not (os.path.exists(AUTH_PATH)):
            sleep(1)
            if (os.path.isdir(DRIVE_PATH)):
                if (os.path.isdir(FAIL_PATH)):
                    PathValid=True
                    validate_usb_write()
                    PathValid=False
                    print "To finish path correction, remove and replace USBDRIVE."

    print "'USBDRIVE' with 'validate.txt' file found."



"""############################################################################

    Function:       set_path_invalid
    Description:    This function is called whenever a path verification check
    has failed in order to inform the user of the error, and to set the
    PathValid variable to false to prevent any further attempts to write to
    the usb drive until the error condition is corrected.

############################################################################"""


def set_path_invalid():

    global PathValid, dataIndex

    # if the path is already defined as invalid, do nothing, else, set the
    # path as invalid, inform the user, and reset the dataIndex.

    if not PathValid == False:
        PathValid = False
        print "USB path is not valid, corrupted, missing, or ejected"
        dataIndex = 0


"""############################################################################

    Function:       validate_usb_write
    Description:    This function checks for the creation of an erroneous USB
    drive path, which can occur rarely if the USB drive is removed, without
    ejecting it first, and the removal occurs between the code that checks
    for a valid path, and where the data is written to the USB drive. If the
    USB doesn't exist, and the write occurs, a 'fake" USBDRIVE owned by root
    will be created and data logging will continue to that location, and all
    future insertions of the USB will be assigned USBDRIVE1, and data will not
    be written to the USB drive again. When this condition is detected, the
    condition will be automatically corrected by erasing the fake path.

    IMPORTANT: to detect this condition a text file called "validate.txt"
    needs to be placed on the USB drive.

############################################################################"""


def validate_usb_write():

    global PathValid, DRIVE_PATH, AUTH_PATH

    # if we already know the path isn't valid, this check isn't needed.
    if PathValid == False:
        return

    # if the "validate.txt" file is not found, then we may have created a
    # 'fake' USB drive path accidentally...
    if not (os.path.exists(AUTH_PATH)):
        print "path corruption suspected"
        sleep(1)

        # The sleep above is to provide some manner of debouncing, in case
        # the detection error of "validate.txt" was a timing / race condition.
        # After all, the result could be erasing all of our data, so it pays
        # to double-check.

        # if the "validate.txt" file is not found a second time...
        if not (os.path.exists(AUTH_PATH)):
            print "path corruption confirmed"
            print "validate.txt file was not found"
            set_path_invalid()

            # remove the drive path. the location that is being written to is
            # not the USB drive. This happens rarely, when attempting a write
            # to a USB that doesn't exist. Linux will create a temporary
            # location, and will appear to be logging to the USB, but isn't.
            rmtree(DRIVE_PATH, ignore_errors = True)

            # if the path no longer exists then rmtree worked, and the
            # incorrect path was deleted.
            if not (os.path.isdir(DRIVE_PATH)):
                print "path corruption corrected"



"""############################################################################

    Function:       start_new_file
    Description:    This function is called when a new file is needed to save
    data logging results. A filename is assembled based on the current Date
    and Time, and if the creation of the file is successful, the file is
    initialized with a header row. If all was successful, PathValid will be
    set to True (This is the only location where this is set to True)

############################################################################"""


def start_new_file():

    global PathValid, FileName, FilePath, dataIndex

    # Assemble the filename based on the current date and time. Then assign
    # the FilePath to start using the new file.
    FileName = str("_".join(str(datetime.datetime.now()).split(" ")))
    FileName = str("_".join(FileName.split(":")))
    FileName = str("_".join(FileName.split(".")))
    FileName = str("_".join(FileName.split("-")))
    FileName = FileName + ".csv"
    FilePath = DRIVE_PATH + FileName

    # if the drive path exists...
    if (os.path.isdir(DRIVE_PATH)):

        # create a new blank file. If this file existed, which it shouldn't,
        # open with 'w' would overwrite it.
        mFile = open(FilePath, "w")

        # display to the user that a new file has been started
        print
        print "#############################################################"
        print "New File: " + FileName
        print "#############################################################"
        print

        # write to the new file a header row to indicate what each of the
        # values represent. In this example, the 'Data' values are unknown. It
        # is expected that 'Data' is a string of an unknown number of values,
        # to which the proper csv formatting has been applied. If you know your
        # data formatting, and want to make this program specific, add your
        # data headers here.

        mFile.write("Index, Date, Time, Data \r\n")
        mFile.close()
        PathValid = True
        dataIndex = 1

    # if the drive path didn't exist...
    else:

        set_path_invalid();


"""############################################################################

    Function:       read_data
    Description:    This function reads the serial input, assembling chars
    until the new line character is received. On reception of a new line,
    the function generates a timestamp, which is pre-pended to the received
    data, and the entire string is written to the display and logged to the
    usb drive (if a valid drive exists).

############################################################################"""


def read_data():

    global PathValid, FileName, FilePath, dataIndex, ser

    # if there is currently no file to write to, attempt to create a file. If
    # that fails, simply return. This means that there is no valid USB inserted
    # so there is no reason to attempt writing to a file.

    if (PathValid == False):
        start_new_file()
        if (PathValid == False):
            return

    valueString = ""        # reset the string that will collect chars
    mchar = ser.read()      # read the next char from the serial port

    # continue reading characters and appending them to the valueString
    # (ignoring carriage returns) until a 'new line' character is received.

    while (mchar != '\n'):
        if (mchar != '\r'):
            valueString += mchar
        mchar = ser.read()

    # after a full valueString has been assembled, create the timestamp

    millis = int(round(time.time() * 1000))
    rightNow = str(datetime.datetime.now()).split()
    mDate = rightNow[0]
    mTime = rightNow[1]

    # format the full string: index, timestamp, and data

    fileString = str(format('%05d', dataIndex)) + ", " + \
        str(mDate) + ", " + str(mTime) + ", " + valueString


    # if execution reaches this point, then it is assumed that a valid USB is
    # inserted, and a file exists to write the data. Open the data file with
    # 'a' to append the new data, write the data string, close the file, and
    # increment the index.

    try:

        # before writing, double-check that the data logging file exists

        if (os.path.exists(FilePath)):
            print(fileString)
            fileString += "\r\n"
            mFile = open(FilePath, "a", 1)
            mFile.write(fileString)
            mFile.close()
            dataIndex += 1
            validate_usb_write()

        # if the file doesn't exist, but the drive path still exsists, then it
        # is possible that the file was deleted while in use.

        elif (os.path.isdir(DRIVE_PATH)):
            start_new_file()

        else:
            set_path_invalid()

    except:

        print("write failed")

    # if the number of rows written to the data file exceeds MAX_ROWS_IN_FILE
    # start a new file. This was added to address an issue where Excel would
    # not import more than 65535 rows from a csv file, when the csv is opened
    # directly, even though Excel is supposed to have a maximum number of rows
    # closer to 1048576.

    if (dataIndex >= MAX_ROWS_IN_FILE):
        start_new_file()


"""############################################################################

    Function:       main

############################################################################"""


def main():

    global ser

    ser = serial.Serial()
    ser.baudrate = 57600
    ser.timeout = None
    ser.port = '/dev/serial0'

    print ser
    print ser.name
    ser.open()
    print "Serial port is open: ", ser.isOpen()

    chk_usb_on_start()

    start_new_file()

    try:
        while (True):
            read_data()

    finally:
        ser.close
        print "Serial port is open: ", ser.isOpen()

    return 0


"""############################################################################

    This next section checks to see whether the file is being executed
    directly. If it is, then __name__ == '__main__' will evaluate as True, and
    the main() function will execute. If the file is being imported by another
    module, then __name__ will be set to the modules name instead.

############################################################################"""


if __name__ == '__main__':
    main()


"""############################################################################
    End of File: datalogger.py
############################################################################"""



Installing and Testing the Script

Before setting up the automation, try installing and running the script manually. For this initial test, it is recommended that the USB drive be removed, and the serial data stream be disconnected before starting. If everything works, this should provide a nice 'static' display to observe, and if needed troubleshoot. To download and test the datalogger, open a terminal window, and perform the following actions.

Create a folder called datalogger in the pi home directory, and then enter the new directory.

cd ~
mkdir datalogger
cd datalogger

Download the datalogger.py Python script file, by entering the following in the terminal window. The wget command will download the specified file, and place it in your current directory, which is the datalogger directory.

wget http:www.davesmotleyprojects.com/raspi/raspi-data-logger/datalogger.py

Now launch the script manually. Note: sudo is required in order to allow Python to access the Serial UART, which is a restricted resource.

sudo python datalogger.py

If everything worked, you should see the following display. The serial port being used is /dev/serial0, which is a high-performance hardware UART, and when queried the port should respond as 'Serial port is open: True'. If the USB drive was not installed when the datalogger started, a user prompt to install a correctly configured USB drive will also be displayed.

2017-02-19-1487524378_655x225_scrot.png image

If the Serial port isn't opening, verify that the RPi was made "Project Ready". If the Serial port is working but there appears to be something wrong with the script, you may want to consider opening and running the script inside of an editor like geany. You can do that by entering the following. Note: sudo is still required to access the serial port. The '&' at the end simply releases terminal window after executing the command. If you find an issue with the script, please provide feedback, and I will correct it.

sudo geany &

If everything worked above, insert a properly configured USB drive (but don't connect the serial data source yet). The script should detect the existence of the drive, and will indicate that it was found, and a new file will be created and the file name will be displayed, as shown below.

2017-02-19-1487523657_655x330_scrot.png image

Now connect the serial data source. Data logging should start, as shown in the window below.

2017-02-19-1487523696_655x330_scrot.png image

When you open the corresponding .csv file on the USB drive, you will see the following data. (Only the first several lines are shown). The first row is added to the file each time a new file is created. This action is performed in the function start_new_file(). Currently everything beyond "Index, Date, Time" is listed as just "Data" because the script is currently generic. It doesn't know, or care, what data is passed to it or what format the data is in. If the script is customized for a specific purpose, you may want to consider changing the first row to include the actual data headers.

            
Index, Date, Time, Data 
00001, 2017-02-19, 12:01:11.805133, 158,  9E, 10011110  
00002, 2017-02-19, 12:01:12.325190, 159,  9F, 10011111  
00003, 2017-02-19, 12:01:12.845139, 160,  A0, 10100000  
00004, 2017-02-19, 12:01:13.365236, 161,  A1, 10100001  
00005, 2017-02-19, 12:01:13.885333, 162,  A2, 10100010  
00006, 2017-02-19, 12:01:14.405431, 163,  A3, 10100011  
00007, 2017-02-19, 12:01:14.925677, 164,  A4, 10100100 



Automatically Start Data Logging

Once everything is working manually, modify the startup.sh script to automatically launch the data logging script at power-on.

In the terminal window, enter the following:

    cd ~
    sudo leafpad startup.sh

This will return to the /home/pi directory, and open the startup.sh script. In that file, enter the following items, save the file, and exit.

    #!/bin/bash
    sleep 5
    echo "Starting data logger"
    xterm -fg white -bg black -fa monaco -fs 12 -geometry 90x24 -e "sudo python ./datalogger/datalogger.py"

If they aren't already, insert the USB drive and connect the serial data source, then reboot the RPi by entering the following in the terminal window.

    sudo reboot

If everything works properly, the data logging will start automatically when the RPi restarts.

2017-02-19-1487526276_600x375_scrot.png image


Why USBDRIVE and validate.txt File?

One of the things that may not be obvious, is why the requirement to label the USB drive as 'USBDRIVE', and why must it contain a file called 'validate.txt'?

The answer to the first question is that this allows us to know how the USB drive will be mounted. The file path will be '/media/pi/USBDRIVE'.

Answering the second question is a little more complicated. This has to do with the operating system response to executing the Python open(filepath, "a") command when the USB filepath doesn't exist. This can happen when the operating system un-mounts the drive just before the execution of the open command. In response, Raspbian will create a 'fake' USBDRIVE directory under /media/pi. When this occurs the Python script would normally continue data logging because the directory appears to exist, however the 'fake' USBDRIVE directory is not accessible. When the real USB drive is re-mounted, Raspbian will mount it as USBDRIVE1, and the data logging will be permanently broken.

To resolve this, the 'validate.txt' file requirement was added. The reason is that the 'validate.txt' file will not exist in the 'fake' USBDRIVE directory, and the condition can be detected and resolved by deleting the 'fake' path. This allows the real USB drive to be re-mounted correctly as USBDRIVE and data logging continues properly when re-inserted. The screenshot below shows the response when the 'fake' path is created by continuing to remove and insert the USB drive.

2017-02-19-1487526886_655x300_scrot.png image


Data Logging File Write Speed

In order to keep the file size manageable, a new file is created each time the number of rows in the file reaches 65535. The file size is controlled by the value of MAX_ROWS_IN_FILE. An interesting question is, does the write speed slow down when appending to files that are very large? In order to test this, a small Python script was used to write 100000 lines to a text file on the USB drive, tracking the time before and immediately after creating the time stamp and data string, printing to the display, and opening, appending, and closing the USB text file. A plot of the resulting write speed is shown below. Although, it can be seen that occasionally the operating system will execute other tasks, interrupting the Python script, and extending the time, there is no apparent trend to suggest that write speed is dependent on file size.

The average write speed is about 4.5 ms.

write speed image