MicroPython release 1.20.2.rc2 based on Espressif IDF 3.3.1 (#394)

* BLE: Allow static passwords, remove bonded when pin changes

* mods/pybadc.c: Fix the argument handling of bits=x for adc.init()

Before, the only accepted argument & value for adc.init() was bits=12, and even that was
discarded. With that change, the function meets the documentation and works as expected.

* mods/pyblte.c: Fix the lenght field for AT commands

At three places the length field was not set when AT commands werde
issued, causing lte.attach() with the band=xx option to fail.
Also, affected: lte_get_modem_version()

* Support Sigfox registration for Pybytes

* PYFW-390: Update .gitignore to allow esp32/lib libraries to be updated

* MDNS advertisement works

* Adding text to advertisement works

* Query works

* Free up internal query results

* Change host_name to hostname

* Rename text to txt

* Adding libmdns.a to esp32/lib

* Adding scripts for PyJTAG and short Readme

* update sigfox libraries

* Replace xQueueSendFromISR call with xQueueSend when called from BLE event

Use the callback queue to notify the LoRa Timer Thread
Do context switch in TimerCallback only if needed

* Replacing "switch-case" with "if-else if" in Region.c to avoid generated Assembly instructions which are reading the Flash from IRAM functions causing bad00bad issue

* BLE characteristic update messages are lost if they sent too frequently

In modbt.c it can happen that received packets are lost, because the user registered callback is not called fast enough (via gatts_char_callback_handler()), which means it can happen that the value of a characteristic is updated 2 times in a row (via gatts_event_handler()) without giving the possibility to the callback to run and fetch the new value. In this specific example the first callback will show the 2nd change, and the 2nd callback will show that actually nothing happened. The 1st change gets lost.
The fix of this issue breaks backward compatibility, because the prototype of the user registered Python callback is changed:
OLD: callback(characteristic)
NEW: callback(characteristic, (event, value))  → (event, value) is a tuple, where data is bytearray containing the value of the request if the “event” is WRITE, otherwise “value” is “None”. “event” shows the correct event any time.

* PYFW-394: mod_ssl_setup_socket() allocates memory while GIL is not locked

* PYFW-401: Mutex of LFS can be locked in wlan_read_file or mod_ssl_read_file

* PYFW-402: wlan_do_connect() allocates memory while GIL is not locked

PYFW-403: mod_network_register_nic() using MicroPython functions outside of GIL in wlan_setup_ap()

* PYFW-404: bt_connect_helper() uses MicroPython functions outside of GIL

* PYFW-405: lte_send_at_cmd() uses MicroPython functions outside of GIL

* refactor lte_add_band() out of lte_attach()

* add bands=() parameter to lte.attach()

this way the user can select a set of bands they want to use when attaching

* PYFW-391: Add changes needed for esp-idf v3.3.1

* Updating README.md

* Update sdkconfig.h and libraries when secure boot is ON, but CONFIG_SECURE_BOOT_ENABLED is removed from sdkconfig.h

* Add checker which forbids compilation when CONFIG_SECURE_BOOT_ENABLED is defined but SECURE=on is not

* Add pybytes_on_boot functionality (#87)

* Update pycom_version.h

Update version to 1.20.2.rc2 for public release

* Update Pybytes to version 1.3.1

Co-authored-by: oligauc <oliver@pycom.io>
Co-authored-by: Islam Wahdan <islam@pycom.io>
Co-authored-by: robert-hh <robert@hammelrath.com>
Co-authored-by: Géza Husi <husigeza91@gmail.com>
Co-authored-by: Jirka Krepl <jiri.krepl@gmail.com>
Co-authored-by: doniks <p.putz@yahoo.de>

Author: 7 people

Jenkinsfile

@@ -9,7 +9,7 @@ node {
     stage('Checkout') {
         checkout scm
         sh 'rm -rf esp-idf'
-        sh 'git clone --depth=1 --recursive -b idf_v3.2 https://github.com/pycom/pycom-esp-idf.git esp-idf'
+        sh 'git clone --depth=1 --recursive -b idf_v3.3.1 https://github.com/pycom/pycom-esp-idf.git esp-idf'
     }
 
     stage('git-tag') {

README.md

@@ -26,7 +26,7 @@ board (PyBoard), the officially supported reference electronic circuit board.
 The following components are actively maintained by Pycom:
 - py/ -- the core Python implementation, including compiler, runtime, and
   core library.
-- exp32/ -- a version of MicroPython that runs on the ESP32 based boards from Pycom.
+- esp32/ -- a version of MicroPython that runs on the ESP32 based boards from Pycom.
 - tests/ -- test framework and test scripts.
 
 Additional components:
@@ -74,7 +74,7 @@ Then when you need the toolchain you can type ``get_esp32`` on the command line
 You also need the ESP IDF along side this repository in order to build the ESP32 port.
 To get it:
 
-    $ git clone --recursive -b idf_v3.2 https://github.com/pycom/pycom-esp-idf.git
+    $ git clone --recursive -b idf_v3.3.1 https://github.com/pycom/pycom-esp-idf.git
 
 After cloning, if you did not specify the --recursive option, make sure to checkout all the submodules:
 

esp32/Makefile

@@ -14,7 +14,7 @@ ifeq ($(wildcard boards/$(BOARD)/.),)
     $(error Invalid BOARD specified)
 endif
 
-IDF_VERSION=3.2
+IDF_VERSION=3.3.1
 
 TARGET ?= boot_app
 
@@ -103,17 +103,10 @@ LIBS = -L$(ESP_IDF_COMP_PATH)/esp32/lib -L$(ESP_IDF_COMP_PATH)/esp32/ld -L$(ESP_
        $(ESP_IDF_COMP_PATH)/newlib/lib/libc-psram-workaround.a \
        -lfreertos -ljson -ljsmn -llwip -lnewlib -lvfs -lopenssl -lmbedtls -lwpa_supplicant \
        -lxtensa-debug-module -lbt -lsdmmc -lsoc -lheap -lbootloader_support -lmicro-ecc \
-       -u ld_include_panic_highint_hdl -lsmartconfig_ack -lmesh -lesp_ringbuf -lcoap
-ifeq ($(BOARD), $(filter $(BOARD), FIPY))
-    LIBS += sigfox/modsigfox_fipy.a
-endif
-
-ifeq ($(BOARD), $(filter $(BOARD), LOPY4))
-    LIBS += sigfox/modsigfox_lopy4.a
-endif
-
-ifeq ($(BOARD), $(filter $(BOARD), SIPY))
-    LIBS += sigfox/modsigfox_sipy.a
+       -u ld_include_panic_highint_hdl -lsmartconfig_ack -lmesh -lesp_ringbuf -lcoap -lmdns -lefuse -lespcoredump -lapp_update
+ifeq ($(BOARD), $(filter $(BOARD), SIPY LOPY4 FIPY))
+    LIBS += sigfox/modsigfox_$(BOARD).a
+$(BUILD)/application.elf: sigfox/modsigfox_$(BOARD).a
 endif
 
 ifeq ($(OPENTHREAD), on)
@@ -134,7 +127,7 @@ endif #ifeq ($(LTE_LOG_BUFF),1)
 
 B_LIBS = -Lbootloader/lib -Lbootloader -L$(BUILD)/bootloader -L$(ESP_IDF_COMP_PATH)/esp32/ld \
          -L$(ESP_IDF_COMP_PATH)/esp32/lib -llog -lcore -lbootloader_support \
-         -lspi_flash -lsoc -lmicro-ecc -lgcc -lstdc++ -lgcov
+         -lspi_flash -lsoc -lmicro-ecc -lgcc -lstdc++ -lgcov -lefuse
 
 # objcopy paramters, to transform a binary file into an object file
 OBJCOPY_EMBED_ARGS = --input-target binary --output-target elf32-xtensa-le --binary-architecture xtensa --rename-section .data=.rodata.embedded

esp32/PyJTAG/Readme.md

@@ -0,0 +1,57 @@
+# Short readme for how to use the PyJTAG
+
+Create the firmware with `BTYPE=debug` flag.
+Do not use the default pins assigned to UART, SPI, CAN because they are used by the JTAG. Pins not to be used: P4, P9, P10, P23
+
+Detailed information are here: https://pycomiot.atlassian.net/wiki/spaces/FIR/pages/966295564/Usage+of+PyJTAG
+
+Setup the PyJTAG board's switches:
+ * ESP32 JTAG: all turned ON
+ * ESP32 B.LOADER: all turned ON except SAFE_BOOT_SW which is OFF
+ * TO LTE UART 1/2: does not matter
+ * CURRENT SHUNTS: connected
+
+Place the Pycom board with the reset button towards the Current Shunts.
+
+Generally follow these rules to setup JTAG debugging on your OS: https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/jtag-debugging/index.html
+
+(Download link of OpenOCD for ESP32 from Espressif: https://github.com/espressif/openocd-esp32/releases)
+
+Connect the PyJTAG via usb. You see four new USB devices:
+```
+$ lsusb -d 0403:
+Bus 001 Device 010: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
+$ ls /dev/ttyUSB?
+/dev/ttyUSB0  /dev/ttyUSB1  /dev/ttyUSB2  /dev/ttyUSB3
+```
+
+Go to `esp32` folder in Firmware-Development repository and run:
+```
+PATH_TO_OPENOCD/bin/openocd -s PATH_TO_OPENOCD/share/openocd/scripts -s PyJTAG -f PyJTAG/interface/ftdi/esp32-pycom.cfg -f PyJTAG/board/esp32-pycom.cfg
+```
+
+Output should be like: 
+```
+Open On-Chip Debugger  v0.10.0-esp32-20191114 (2019-11-14-14:15)
+Licensed under GNU GPL v2
+For bug reports, read
+	http://openocd.org/doc/doxygen/bugs.html
+none separate
+adapter speed: 20000 kHz
+Info : Configured 2 cores
+Info : Listening on port 6666 for tcl connections
+Info : Listening on port 4444 for telnet connections
+Error: type 'esp32' is missing virt2phys
+Info : ftdi: if you experience problems at higher adapter clocks, try the command "ftdi_tdo_sample_edge falling"
+Info : clock speed 20000 kHz
+Info : JTAG tap: esp32.cpu0 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
+Info : JTAG tap: esp32.cpu1 tap/device found: 0x120034e5 (mfg: 0x272 (Tensilica), part: 0x2003, ver: 0x1)
+Info : Listening on port 3333 for gdb connections
+```
+
+When OpenOCD is running, start GDB from `esp32` folder. Assuming you have a FIPY:
+```
+xtensa-esp32-elf-gdb -x PyJTAG/gdbinit build/FIPY/debug/application.elf
+```
+
+In `PyJTAG/gdbinit` a breakpoint is configured at `TASK_Micropython`, so execution should stop there first.

esp32/PyJTAG/board/esp32-pycom.cfg

@@ -0,0 +1,2 @@
+set ESP32_FLASH_VOLTAGE 3.3
+source [find target/esp32-pycom.cfg]

esp32/PyJTAG/gdbinit

@@ -0,0 +1,5 @@
+target remote :3333
+mon reset halt
+flushregs
+thb TASK_Micropython
+c

esp32/PyJTAG/interface/ftdi/esp32-pycom.cfg

@@ -0,0 +1,35 @@
+#
+# Driver for the FT4232HL JTAG chip on the Pycom's PyJTAG board
+#
+ 
+
+interface ftdi
+ftdi_vid_pid 0x0403 0x6011
+
+# interface 1 is the uart
+ftdi_channel 0
+
+# TCK, TDI, TDO, TMS: ADBUS0-3
+# LEDs: ACBUS4-7
+
+ftdi_layout_init 0x0008 0xf00b
+#ftdi_layout_signal LED -data 0x1000
+#ftdi_layout_signal LED2 -data 0x2000
+#ftdi_layout_signal LED3 -data 0x4000
+#ftdi_layout_signal LED4 -data 0x8000
+
+# ESP32 series chips do not have a TRST input, and the SRST line is connected
+# to the EN pin. 
+# The target code doesn't handle SRST reset properly yet, so this is
+# commented out:
+# ftdi_layout_signal nSRST -oe 0x0020
+
+reset_config none
+
+# The speed of the JTAG interface, in KHz. If you get DSR/DIR errors (and they
+# do not relate to OpenOCD trying to read from a memory range without physical
+# memory being present there), you can try lowering this.
+#
+# On DevKit-J, this can go as high as 20MHz if CPU frequency is 80MHz, or 26MHz
+# if CPU frequency is 160MHz or 240MHz.
+adapter_khz 20000

esp32/PyJTAG/target/esp32-pycom.cfg

@@ -0,0 +1,69 @@
+# The ESP32 only supports JTAG.
+transport select jtag
+
+# Source the ESP common configuration file
+source [find target/esp_common.cfg]
+
+if { [info exists CHIPNAME] } {
+	set _CHIPNAME $CHIPNAME
+} else {
+	set _CHIPNAME esp32
+}
+
+if { [info exists CPUTAPID] } {
+	set _CPUTAPID $CPUTAPID
+} else {
+	set _CPUTAPID 0x120034e5
+}
+
+if { [info exists ESP32_ONLYCPU] } {
+	set _ONLYCPU $ESP32_ONLYCPU
+} else {
+	set _ONLYCPU 3
+}
+
+if { [info exists ESP32_FLASH_VOLTAGE] } {
+	set _FLASH_VOLTAGE $ESP32_FLASH_VOLTAGE
+} else {
+	set _FLASH_VOLTAGE 3.3
+}
+
+set _TARGETNAME $_CHIPNAME
+set _CPU0NAME cpu0
+set _CPU1NAME cpu1
+set _TAPNAME $_CHIPNAME.$_CPU0NAME
+
+jtag newtap $_CHIPNAME $_CPU0NAME -irlen 5 -expected-id $_CPUTAPID
+if { $_ONLYCPU != 1 } {
+	jtag newtap $_CHIPNAME $_CPU1NAME -irlen 5 -expected-id $_CPUTAPID
+} else {
+	jtag newtap $_CHIPNAME $_CPU1NAME -irlen 5 -disable -expected-id $_CPUTAPID
+}
+
+if { $_RTOS == "none" } {
+	target create $_TARGETNAME esp32 -endian little -chain-position $_TAPNAME
+} else {
+	target create $_TARGETNAME esp32 -endian little -chain-position $_TAPNAME -rtos $_RTOS
+}
+
+configure_esp_workarea $_TARGETNAME 0x40090000 0x3400 0x3FFC0000 0x6000
+configure_esp_flash_bank $_TARGETNAME $_TARGETNAME $_FLASH_SIZE
+
+esp32 flashbootstrap $_FLASH_VOLTAGE
+esp32 maskisr on
+if { $_SEMIHOST_BASEDIR != "" } {
+	esp32 semihost_basedir $_SEMIHOST_BASEDIR
+}
+if { $_FLASH_SIZE == 0 } {
+	gdb_breakpoint_override hard
+}
+
+# special function to program ESP32, it differs from the original 'program' that
+# it verifies written image by reading flash directly, instead of reading memory mapped flash regions
+proc program_esp32 {filename args} {
+	program_esp $filename $args
+}
+
+add_help_text program_esp32 "write an image to flash, address is only required for binary images. verify, reset, exit are optional"
+add_usage_text program_esp32 "<filename> \[address\] \[verify\] \[reset\] \[exit\]"
+

esp32/application.mk

@@ -77,6 +77,7 @@ APP_INC += -I$(ESP_IDF_COMP_PATH)/coap/libcoap/include/coap
 APP_INC += -I$(ESP_IDF_COMP_PATH)/coap/libcoap/examples
 APP_INC += -I$(ESP_IDF_COMP_PATH)/coap/port/include
 APP_INC += -I$(ESP_IDF_COMP_PATH)/coap/port/include/coap
+APP_INC += -I$(ESP_IDF_COMP_PATH)/mdns/include
 APP_INC += -I../lib/mp-readline
 APP_INC += -I../lib/netutils
 APP_INC += -I../lib/oofatfs
@@ -160,6 +161,7 @@ APP_MODS_SRC_C = $(addprefix mods/,\
 	lwipsocket.c \
 	machtouch.c \
 	modcoap.c \
+	modmdns.c \
 	)
 
 APP_MODS_LORA_SRC_C = $(addprefix mods/,\
@@ -183,6 +185,7 @@ APP_UTIL_SRC_C = $(addprefix util/,\
 	mpsleep.c \
 	timeutils.c \
 	esp32chipinfo.c \
+	pycom_general_util.c \
 	)
 
 APP_FATFS_SRC_C = $(addprefix fatfs/src/,\
@@ -245,7 +248,7 @@ APP_SX1276_SRC_C = $(addprefix drivers/sx127x/,\
 	sx1276/sx1276.c \
 	)
 
-APP_SIGFOX_SRC_SIPY_C = $(addprefix sigfox/,\
+APP_SIGFOX_SRC_SIPY_C = $(addprefix sigfox/src/,\
 	manufacturer_api.c \
 	radio.c \
 	ti_aes_128.c \
@@ -254,7 +257,7 @@ APP_SIGFOX_SRC_SIPY_C = $(addprefix sigfox/,\
 	modsigfox.c \
 	)
 
-APP_SIGFOX_SRC_FIPY_LOPY4_C = $(addprefix sigfox/,\
+APP_SIGFOX_SRC_FIPY_LOPY4_C = $(addprefix sigfox/src/,\
 	manufacturer_api.c \
 	radio_sx127x.c \
 	ti_aes_128.c \
@@ -267,7 +270,7 @@ APP_SIGFOX_MOD_SRC_C = $(addprefix mods/,\
 	modsigfox_api.c \
 	)
 
-APP_SIGFOX_TARGET_SRC_C = $(addprefix sigfox/targets/,\
+APP_SIGFOX_TARGET_SRC_C = $(addprefix sigfox/src/targets/,\
 	cc112x_spi.c \
 	hal_int.c \
 	hal_spi_rf_trxeb.c \
@@ -366,7 +369,7 @@ SRC_QSTR_AUTO_DEPS +=
 BOOT_LDFLAGS = $(LDFLAGS) -T esp32.bootloader.ld -T esp32.rom.ld -T esp32.peripherals.ld -T esp32.bootloader.rom.ld -T esp32.rom.spiram_incompatible_fns.ld
 
 # add the application linker script(s)
-APP_LDFLAGS += $(LDFLAGS) -T esp32_out.ld -T esp32.common.ld -T esp32.rom.ld -T esp32.peripherals.ld -T wifi_iram.ld
+APP_LDFLAGS += $(LDFLAGS) -T esp32_out.ld -T esp32.project.ld -T esp32.rom.ld -T esp32.peripherals.ld
 
 # add the application specific CFLAGS
 CFLAGS += $(APP_INC) -DMICROPY_NLR_SETJMP=1 -DMBEDTLS_CONFIG_FILE='"mbedtls/esp_config.h"' -DHAVE_CONFIG_H -DESP_PLATFORM -DFFCONF_H=\"lib/oofatfs/ffconf.h\" -DWITH_POSIX
@@ -512,6 +515,9 @@ endif
 ifeq ($(TARGET), boot_app)
 all: $(BOOT_BIN) $(APP_BIN)
 endif
+ifeq ($(TARGET), sigfox)
+include sigfox.mk
+endif
 .PHONY: all CHECK_DEP
 
 $(info $(VARIANT) Variant) 
@@ -525,6 +531,14 @@ CFLAGS += -DCONFIG_FLASH_ENCRYPTION_ENABLED=1
 # it can also be added permanently in sdkconfig.h
 CFLAGS += -DCONFIG_SECURE_BOOT_ENABLED=1
 
+define resolvepath
+$(abspath $(foreach dir,$(1),$(if $(filter /%,$(dir)),$(dir),$(subst //,/,$(2)/$(dir)))))
+endef
+
+define dequote
+$(subst ",,$(1))
+endef
+
 # find the configured private key file
 ORIG_SECURE_KEY := $(call resolvepath,$(call dequote,$(SECURE_KEY)),$(PROJECT_PATH))
 
@@ -639,7 +653,7 @@ $(BUILD)/application.a: $(OBJ)
 	$(ECHO) "AR $@"
 	$(Q) rm -f $@
 	$(Q) $(AR) cru $@ $^
-$(BUILD)/application.elf: $(BUILD)/application.a $(BUILD)/esp32_out.ld $(SECURE_BOOT_VERIFICATION_KEY)
+$(BUILD)/application.elf: $(BUILD)/application.a $(BUILD)/esp32_out.ld esp32.project.ld $(SECURE_BOOT_VERIFICATION_KEY)
 ifeq ($(SECURE), on)
 # unpack libbootloader_support.a, and archive again using the right key for verifying signatures
 	$(ECHO) "Inserting verification key $(SECURE_BOOT_VERIFICATION_KEY) in $@"
@@ -807,6 +821,7 @@ $(OBJ): | $(GEN_PINS_HDR)
 CHECK_DEP:
 	$(Q) bash tools/idfVerCheck.sh $(IDF_PATH) "$(IDF_VERSION)"
 	$(Q) bash tools/mpy-build-check.sh $(BOARD) $(BTYPE) $(VARIANT)
+	$(Q) $(PYTHON) check_secure_boot.py --SECURE $(SECURE)
 ifeq ($(COPY_IDF_LIB), 1)
 	$(ECHO) "COPY IDF LIBRARIES"
 	$(Q) $(PYTHON) get_idf_libs.py --idflibs $(IDF_PATH)/examples/wifi/scan/build

esp32/bootloader/lib/libbootloader_support.a

@@ -0,0 +1,18 @@
+import argparse
+
+
+def main():
+    cmd_parser = argparse.ArgumentParser()
+    cmd_parser.add_argument('--SECURE', default=None)
+    cmd_args = cmd_parser.parse_args()
+
+    secure = cmd_args.SECURE
+    with open("sdkconfig.h") as sdkconfig:
+        if any("CONFIG_SECURE_BOOT_ENABLED" in l for l in sdkconfig.readlines()):
+            if(secure != "on"):
+                print("If CONFIG_SECURE_BOOT_ENABLED is defined in sdkconfig.h, the SECURE=on must be used when building the Firmware!")
+                # Non zero exit code means error
+                exit(1)
+
+if __name__ == "__main__":
+    main()