Certification Automation Setup

To automate certification tests, additional setup is required.

All installation and configuration steps below occur on the Windows machine running the GRL Test Harness software. Download and install the latest GRL Thread Test Harness before continuing.

Once installed, there are two automation methods for the device under test (DUT):

  1. OpenThread Harness Automation Tool
  2. Test Harness Auto DUT mode

OpenThread Harness Automation Tool

To use this tool, the DUT must be running OpenThread 1.1 with CLI support, and fully controllable by all commands wrapped in /tools/harness-automation/autothreadhardness/open_thread_controller.py.

We recommend using a power distribution unit (PDU) with the Harness Automation Tool, for ease in resetting all devices, but this is not required.

Install

  1. Install the following software:
    • Python 2.7
    • Git for Windows (Bash support is highly recommended)
  2. Clone the OpenThread repository to get the automation scripts:
    git clone https://github.com/openthread/openthread.git
    
  3. Install Python requirements:
    cd tools/harness-automation
    pip install -r requirements.txt
    
  4. Install both Google Chrome and ChromeDriver.

Configure

  1. Create a configuration file for the automation tool:
    cp autothreadharness/settings_sample.py autothreadharness/settings.py
    
  2. Update that new configuration file according to the instructions in the file.
  3. Add the following parameter to the Test Harness configuration file, found at C:\GRL\Thread1.1\Config\Configuration.ini:
    BrowserAutoNavigate = False

Mixed testbed support

The Harness Automation Tool also supports mixed testbeds, where the reference devices used in each test case are a mixed set of devices rather than all the same device. For example, instead of using 32 TI CC2538 boards running OpenThread for all test cases, you could mix and match between all four Thread reference devices for each test case as desired.

To use mixed testbeds, additional configuration is necessary:

  1. Add the following additional parameter to the Test Harness configuration file, found at C:\GRL\Thread1.1\Config\Configuration.ini:
    EnableDeviceSelection = True
  2. Download a topology configuration file from the Thread Group. Add this file as TopologyConfig.txt to C:\GRL\Thread1.1\Thread_Harness\TestScripts. This file details which reference device to use for each role in each test case.

The topology configuration file provided by the Thread Group can be modified for other mixed bed configurations. Valid values to use for reference devices in the topology configuration file are:

Value in topology configuration file Test Harness reference device
ARM ARM: NXP FRDM-K64F with Firefly 6LoWPAN Shield
NXP NXP(Freescale): USB-KW24D512 Dongles
OpenThread OpenThread: TI CC2538DK
SiLabs Silabs: EM358x USB Stick

Each test case entry in the topology configuration file must be on a separate line and formatted in role:device pairs as follows:

5.3.10-Leader:NXP,Router_1:OpenThread,BorderRouter:OpenThread,MED_1:ARM

Test Harness Auto DUT mode

The GRL Test Harness software provides an advanced auto discovery feature (Auto DUT) which accelerates the validation process for Thread reference stack vendors.

OpenThread provides an example Thread Host Controller Interface (THCI) that allows the Test Harness to control your component or product as if it were one of the reference platforms. According to the serial port characteristics of your specific platform, your DUT could either:

Acting as the existing OpenThread reference platform

If the serial port of the DUT functions under settings of 115200 8-N-1, and the device responds correctly after every physical device reset (for example, to confirm the device's valid connection to the Windows machine), then the platform can use the OpenThread THCI tool. This tool allows the device to act as the OpenThread reference platform during testing.

  1. Close the GRL Test Harness, if it is open
  2. Attach the device to the Windows machine
    OT Certification - Device Manager example
  3. Find the device serial port hardware identifier in Device Manager:
    1. From the Windows command line, open Device Manager:
      mmc devmgmt.msc
    2. Under Ports, right-click the device and select Properties
    3. Select the Details tab
    4. In the Property dropdown, select Hardware Ids
    5. Note the VID and PID values for the device. In the screenshot example, the VID is 1366 and the PID is 1015.
  4. Update the Golden Device list:
    1. Open C:\GRL\Thread1.1\Config\Configuration.ini
    2. Update the OpenThread group in the GoldenDeviceHardwareIds array with the VID and PID of the device:
      GoldenDeviceHardwareIds = {
      'NXP': ['VID:PID=1FC9:0300','VID:PID=15A2:0300','VID:PID=1366:1015'],
      'SiLabs': 'VID:PID=10C4:8A5E',
      'ARM': 'VID:PID=0D28:0204',
      'OpenThread':['VID:PID=10C4:EA60','VID:PID=1366:1015']}
  5. Enable Auto DUT mode

Acting as a new reference platform

If the serial port of the DUT does not function under the proper serial port settings, or the device does not respond correctly after a physical device reset (outputs unreadable code within 3-6 seconds of reset), then OpenThread THCI can be customized to treat the device as a new platform in the Test Harness.

For example, to customize THCI using the Dialog DA15000 platform:

  1. Close the GRL Test Harness, if it is open
  2. Modify /tools/harness-thci/OpenThread.py based on the UART characteristics of the target device. Modifications may differ between target devices. In the case of the Dialog DA15000 platform:

    1. Rename the OpenThread.py file to DA15000.py
    2. Change the first three occurrences of "OpenThread" to "DA15000":

      >> Device : DA15000 THCI
      >> Class : DA15000
      
      class DA15000(IThci):
      
    3. Change the serial port parameters:

      def _connect(self):
          print 'My port is %s' % self.port
          if self.port.startswith('COM'):
              self.handle = serial.Serial(self.port, 115200, timeout=0)
              time.sleep(1)
              self.handle.write('\r\n')
              time.sleep(0.1)
              self._is_net = False
          elif ':' in self.port:
              host, port = self.port.split(':')
              self.handle = socket.create_connection((host, port))
              self.handle.setblocking(0)
              self._is_net = True
          else:
              raise Exception('Unknown port schema')
          self.UIStatusMsg = self.getVersionNumber()
      
    4. Change the sleep time to eliminate the impact of unreadable code output after device reset:

      def powerDown(self):
          """power down the Thread device"""
          print '%s call powerDown' % self.port
          self._sendline('reset')
          self.isPowerDown = True
          time.sleep(8)  # New addition
      
      def reboot(self):
          """reset and rejoin to Thread Network without any timeout
      
          Returns:
              True: successful to reset and rejoin the Thread Network
              False: fail to reset and rejoin the Thread Network
          """
          print '%s call reboot' % self.port
          try:
              self._sendline('reset')
              self.isPowerDown = True
              time.sleep(8)  # Updated from 3 to 8
      
      def reset(self):
          """factory reset"""
          print '%s call reset' % self.port
          try:
              self._sendline('factoryreset')
              time.sleep(8)  # Updated from 3 to 8
              self._read()
      
      def resetAndRejoin(self, timeout):
          """reset and join back Thread Network with a given timeout delay
      
          Args:
              timeout: a timeout interval before rejoin Thread Network
      
          Returns:
              True: successful to reset and rejoin Thread Network
              False: fail to reset and rejoin the Thread Network
          """
          print '%s call resetAndRejoin' % self.port
          print timeout
          try:
              self._sendline('reset')
              self.isPowerDown = True
              time.sleep(timeout)
              if timeout < 8:      # Sleep a bit longer if the timeout is short
                  time.sleep(8 - timeout)
      
  3. Copy the modified DA15000.py file to C:\GRL\Thread1.1\Thread_Harness\THCI

  4. Add the new platform information to the Test Harness:

    1. Create an icon (in png or jpg format) to make it easier to differentiate your platform and copy it to C:\GRL\Thread1.1\Web\images
    2. Update C:\GRL\Thread1.1\Web\data\deviceInputFields.xml with a new DEVICE section, where the thumbnail parameter is the icon file:

      <DEVICE name="DA15000" thumbnail="DA15000.jpg" description ="DA15000: Dialog" THCI="DA15000">
          <ITEM label="Serial Line"
                type="text"
                forParam="SerialPort"
                validation="COM"
                hint="eg: COM1">COM
          </ITEM>
          <ITEM label="Speed"
                type="text"
                forParam="SerialBaudRate"
                validation="baud-rate"
                hint="eg: 115200">115200
          </ITEM>
      </DEVICE>
      
  5. Attach the device to the Windows machine

    OT Certification - Device Manager example

  6. Find the device serial port hardware identifier in Device Manager:

    1. From the Windows command line, open Device Manager:
      mmc devmgmt.msc
    2. Under Ports, right-click the device and select Properties
    3. Select the Details tab
    4. In the Property dropdown, select Hardware Ids
    5. Note the VID and PID values for the device
  7. Update the Golden Device list:

    1. Open C:\GRL\Thread1.1\Config\Configuration.ini
    2. Add a new platform group in the GoldenDeviceHardwareIds array with the VID and PID of the device:
      GoldenDeviceHardwareIds = {
      'NXP': ['VID:PID=1FC9:0300','VID:PID=15A2:0300','VID:PID=1366:1015'],
      'SiLabs': 'VID:PID=10C4:8A5E',
      'ARM': 'VID:PID=0D28:0204',
      'OpenThread':'VID:PID=10C4:EA60',
      'DA15000': 'VID:PID=1366:1015'}
  8. Enable Auto DUT mode

Enable Auto DUT mode

After one of the two above configuration options are complete:

  1. Open the GRL Test Harness, the device/port now appears as a new reference device
  2. Select the Enable Auto DUT Device Selection checkbox below the Supported Hardware list
  3. Select the Set as DUT radio button under the target device/port to set the device as the DUT
OT Certification Auto DUT