Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrading to nonos SDK 3.10 #212

Open
scargill opened this issue Feb 21, 2019 · 174 comments
Open

Upgrading to nonos SDK 3.10 #212

scargill opened this issue Feb 21, 2019 · 174 comments

Comments

@scargill
Copy link

I've not upgraded since NONOS SDK 2.1 due to not enough IRAM in later versions. I use RBOOT by Richard Burton - virtually all my functions are in Flash - my code works perfectly in SDK 2.1 - . I've not kept up with the need to define a partition table. Does the latest version (today - 21/02/2019) free up more space and do the docs now simply explain partition table information?
@fvpalha
Copy link

fvpalha commented Feb 21, 2019

I'm using RBOOT, too.
I am using NON SDK 2.2.x commit 34f1abd (12/27/2018).
I'm still not sure on upgrading to the NONOS SDK 3.1.0

The project esp_nano_httpd_fota can be an alternative to RBOOT.

@marcelstoer
Copy link
Contributor

I haven't seen a v3.1 in the wild yet. Pointer?

@scargill
Copy link
Author

scargill commented Feb 21, 2019 via email

@marcelstoer
Copy link
Contributor

Never mind, I thought you were referring to an official release. Seems you just mean the latest commit on master - which was first labeled as v3.1 on Jan 2nd.

@scargill
Copy link
Author

scargill commented Feb 21, 2019 via email

@davydnorris
Copy link

I've been using latest since it came out - for me moving to the partition table model was only a minor hassle in terms of updating my code and my Makefile, but there are several things I have noticed that are quite cool:

  • my flash map is now explicit. I can say exactly how much room my user flash is allowed and can check at build and OTA update time
  • my SSL certificate locations are also explicit and I now use the partition table functions to locate them when needed
  • my user data partition is explicitly defined as well

Truth be told, these things were explicit before, but they were all defined solely in my makefile, and I had separate definitions for the various flash sizes I had to deal with. It's now laid out in my code, and automatically calculated from a single flash size define passed in during the build. I use additional macro directives in the code to output information about where to flash the various items, but these are also in my makefile

My code is now pretty much independent of the makefile settings

@scargill
Copy link
Author

scargill commented Feb 21, 2019 via email

@someburner
Copy link

While I think it's important to actually understand the partition table, which is really very simple- it's just a short list mapping addresses to various sectors of flash needed by the SDK- I can see it being a little confusing if you don't already have these locations defined in your project.

For a 4MB ESP8266 (eg ESP-12E) and rBoot with 512KB roms, all you have to do is, in user_main.c, make it look like this:

#define SDK_RF_CAL_ADDR 0x3FB000
#define SDK_PHY_DATA_ADDR 0x3FC000
#define SDK_PARAM_ADDR 0x3FD000
#define SPI_FLASH_SIZE_MAP  FLASH_SIZE_32M_MAP_512_512 // 4
#define SDK_PRIV_PARAM_ADDR 0x7C000
#define SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM SYSTEM_PARTITION_CUSTOMER_BEGIN

static const partition_item_t p_table[] = {
    { SYSTEM_PARTITION_BOOTLOADER,          0x0,                 0x1000 },
    { SYSTEM_PARTITION_RF_CAL,              SDK_RF_CAL_ADDR,     0x1000 },
    { SYSTEM_PARTITION_PHY_DATA,            SDK_PHY_DATA_ADDR,   0x1000 },
    { SYSTEM_PARTITION_SYSTEM_PARAMETER,    SDK_PARAM_ADDR,      0x3000 },
    { SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM, SDK_PRIV_PARAM_ADDR, 0x1000 },
};

void user_pre_init(void)
{
    if(!system_partition_table_regist(p_table, sizeof(p_table)/sizeof(p_table[0]), SPI_FLASH_SIZE_MAP)) {
        os_printf("system_partition_table_regist fail\r\n");
        while(1);
    }
}

@davydnorris
Copy link

davydnorris commented Feb 22, 2019 •
edited
Loading

Actually @someburner, there are a few things you would need to change:

  • if you're using rBoot, you need two sectors for the boot ROM so the first line needs to be changed to reflect the larger size:
    { SYSTEM_PARTITION_BOOTLOADER, 0x0, 0x2000 },

and you also need to include the actual code partitions in this list, which for an OTA enabled 4MB 512k system would start at 0x002000 an 0x082000 (0x2000 higher than the base OTA due to rBoot), and would be 512k in size, minus 2k for rBoot and minus whatever size needed for your system or your own parameters, whichever is larger. In this case let's assume the system parameters are largest size at 1+1+3=5k, so our max OTA partition size is 0x80000 - 0x002000 - 0x005000 = 0x079000

#define SYSTEM_PARTITION_OTA_SIZE      0x79000

{ SYSTEM_PARTITION_OTA_1,              0x002000,     SYSTEM_PARTITION_OTA_SIZE },
{ SYSTEM_PARTITION_OTA_2,              0x082000,     SYSTEM_PARTITION_OTA_SIZE },

@someburner
Copy link

someburner commented Feb 22, 2019 •
edited
Loading

@davydnorris

Good catch re: rBoot size. Although technically the rboot binary only takes a single sector, the 2nd sector stores configuration.

Regarding SYSTEM_PARTITION_OTA_N parameters, it was my understanding that is only required if using the Espressif OTA method, which I'm not even sure works for rBoot? The configuration I posted boots and OTAs using rBoot-style OTA. It would be nice to know what's going on under the hood and how much these things matter.

@davydnorris
Copy link

I think you may be right there regarding the rBoot OTA approach. I don't know if there's any way you can specify that info for rBoot (I seem to remember seeing something similar though).

I've found having the explicit OTA partition size has saved me once already when I accidentally set up an OTA flash built for a 1M partition for a set of devices that were configured as 512k

@scargill
Copy link
Author

scargill commented Feb 22, 2019 via email

@scargill
Copy link
Author

scargill commented Feb 22, 2019 •
edited
Loading

Which is the correct version - assuming 4MB ESP12 and roms that are in between 512k and 1 meg each... I think I'm up to 700k.??? Your feedback, guys, is MOST appreciated... so that's it? a change to user_main.c?

I can feel success is almost within my grasp...

@someburner
Copy link

someburner commented Feb 22, 2019 •
edited
Loading

If you look in the SDK's include/user_interface.h you can find the flash_size_map enum that you need to look at to choose for the #define SPI_FLASH_SIZE_MAP definition. So if you have roms > 512KB, you will need to use the 1MB size, which would be FLASH_SIZE_32M_MAP_1024_1024, so the definition would be:

#define SPI_FLASH_SIZE_MAP FLASH_SIZE_32M_MAP_1024_1024

and you'd also want to change the SDK_PRIV_PARAM_ADDR so the SDK param sector is at the end of the 1MB, so SDK_PRIV_PARAM_ADDR would become (I think):

#define SDK_PRIV_PARAM_ADDR 0xFC000

@scargill
Copy link
Author

NICE - but I had a feeling it would not be THAT simple - so - I added the code above, adjusted for 1 meg roms - and upgraded to the SDK here in GIT. NOPE.

In SDK 2.1 I get
build/app_0.out: file format elf32-xtensa-le

Sections:
Idx Name Size VMA LMA File off Algn
0 .data 00000d38 3ffe8000 3ffe8000 000000e0 24
CONTENTS, ALLOC, LOAD, DATA
1 .rodata 00002ea8 3ffe8d40 3ffe8d40 00000e20 24
CONTENTS, ALLOC, LOAD, READONLY, DATA
2 .bss 0000b2e0 3ffebbe8 3ffebbe8 00003cc8 24
ALLOC
3 .text 00007774 40100000 40100000 00003cc8 22
CONTENTS, ALLOC, LOAD, READONLY, CODE
4 .irom0.text 0009d284 40202010 40202010 0000b440 2**4
CONTENTS, ALLOC, LOAD, READONLY, CODE

As you can see above compilation works in 2.1 and I still have .text room to go.

Using the new SDK 3.1 - with the additional info as advised above, I get all the way to the end of the compilation then the dreaded error -

c:/espressif/xtensa-lx106-elf/bin/../lib/gcc/xtensa-lx106-elf/4.8.2/../../../../xtensa-lx106-elf/bin/ld.exe: build/app_0.out section .text' will not fit in region iram1_0_seg'
collect2.exe: error: ld returned 1 exit status
mingw32-make.exe: *** [build/app_0.out] Error 1

Just as I've had with previous attempts to upgrade the SDK. Ideas, anyone???

@scargill
Copy link
Author

One more change to the above - I changed the bootloader size to 0x2000 as davydnorris pointed out... made no difference - same ".text will nor fit into region iram1_0_seg" error.

@davydnorris
Copy link

I am happy to take a look at your project @scargill and see if I can get it to fit?

@scargill
Copy link
Author

Here it is: https://bitbucket.org/scargill/esp-go/src/master/ - this one is up to date -- works with SDK 2.1 - the new info in user_main.c is commented out as it prevents compiling for 2.1 - the main make file is in the outer directory - simply called MAKEFILE - th emain file which calls others is /user/petes_code.c

The project is written in C, not using Arduino.... and originally emanated from an MQTT demo and now conrtols a range of peripherals by serial of MQTT.

I'm in the process of adding more fonts.

Pete

@davydnorris
Copy link

OK I've pulled the code and had a bit of a go - had to fiddle a lot with the Makefile, and pull down esptool2 and make that as well, and play with your python script to make Python 3.7 like it, and then I started building...

By default I turn on all warnings and there's a lot to fix - lots of implicit function declarations, which can make the code not optimise very well, and a bunch of other dodgy assignments to and from int32 and pointers. I'll have a bit more of a play but it's taken a chunk of my time already.

@scargill
Copy link
Author

scargill commented Feb 24, 2019 via email

@davydnorris
Copy link

davydnorris commented Feb 24, 2019 •
edited
Loading

I had to change it just to get it to compile - trying not to change anything at all if I can help it. One thing I am doing is pulling the espmissingincludes.h header as they have fixed all that up now. But the rest is just missing headers and definitions.

I also have to get used to rBoot and its link map too

@fvpalha
Copy link

fvpalha commented Feb 28, 2019

Hi.
Someone have a simple example with rboot in SKD 3.1.0 ?

@scargill
Copy link
Author

scargill commented Feb 28, 2019 via email

@fvpalha
Copy link

fvpalha commented Feb 28, 2019

I compiled in SDK 3.1.0 after add the code:

#include "user_interface.h"

#define SDK_RF_CAL_ADDR 0x3FB000
#define SDK_PHY_DATA_ADDR 0x3FC000
#define SDK_PARAM_ADDR 0x3FD000
#define SPI_FLASH_SIZE_MAP  FLASH_SIZE_32M_MAP_512_512 // 4
#define SDK_PRIV_PARAM_ADDR 0x7C000
#define SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM SYSTEM_PARTITION_CUSTOMER_BEGIN

static const partition_item_t p_table[] = {
    { SYSTEM_PARTITION_BOOTLOADER,          0x0,                 0x1000 },
    { SYSTEM_PARTITION_RF_CAL,              SDK_RF_CAL_ADDR,     0x1000 },
    { SYSTEM_PARTITION_PHY_DATA,            SDK_PHY_DATA_ADDR,   0x1000 },
    { SYSTEM_PARTITION_SYSTEM_PARAMETER,    SDK_PARAM_ADDR,      0x3000 },
    { SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM, SDK_PRIV_PARAM_ADDR, 0x1000 },
};

void user_pre_init(void)
{
    if(!system_partition_table_regist(p_table, sizeof(p_table)/sizeof(p_table[0]), SPI_FLASH_SIZE_MAP)) {
        os_printf("system_partition_table_regist fail\r\n");
        while(1);
    }
}

But my code does not run after burn the ESP8266.

@scargill
Copy link
Author

scargill commented Feb 28, 2019 via email

@fvpalha
Copy link

fvpalha commented Mar 1, 2019

@davydnorris and @someburner , can you share the Makefile and the main file of the project than works with rBoot in SDK 3.1.0 ?

I do not understand why the rom does not run.

Thank you.

@eriksl
Copy link

eriksl commented Mar 2, 2019 •
edited
Loading

The latest (not officially released) commits from the SDK (master/HEAD) free up a LOT of IRAM. Definitely worth a try.

If you're using rboot, I don't think it's wise to mark it's flash regions as "well-known" SDK partition, like OTA_1 and OTA_2 etc. The SDK code will make assumptions about it, which may not hold. RBOOT is not the same as the SDK OTA mechanism.

This is my partition table (for use with either rboot or without), as an example for use with rboot:

static const partition_item_t partition_items[] =
{
	{	SYSTEM_PARTITION_RF_CAL, 				RFCAL_OFFSET,				RFCAL_SIZE,				},
	{	SYSTEM_PARTITION_PHY_DATA,				PHYDATA_OFFSET,				PHYDATA_SIZE,			},
	{	SYSTEM_PARTITION_SYSTEM_PARAMETER,		SYSTEM_CONFIG_OFFSET,		SYSTEM_CONFIG_SIZE,		},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 0,	USER_CONFIG_OFFSET,			USER_CONFIG_SIZE,		},
#if IMAGE_OTA == 0
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 1,	OFFSET_IRAM_PLAIN,			SIZE_IRAM_PLAIN,		},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 2,	OFFSET_IROM_PLAIN,			SIZE_IROM_PLAIN,		},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 3,	SEQUENCER_FLASH_OFFSET,		SEQUENCER_FLASH_SIZE,	},
#else
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 1,	OFFSET_OTA_BOOT,			SIZE_OTA_BOOT,			},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 2,	OFFSET_OTA_RBOOT_CFG,		SIZE_OTA_RBOOT_CFG,		},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 3,	OFFSET_OTA_IMG_0,			SIZE_OTA_IMG,			},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 4,	OFFSET_OTA_IMG_1,			SIZE_OTA_IMG,			},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 5,	SEQUENCER_FLASH_OFFSET_0,	SEQUENCER_FLASH_SIZE,	},
	{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 6,	SEQUENCER_FLASH_OFFSET_1,	SEQUENCER_FLASH_SIZE,	},
#endif
};

The actual values are supplied from the Makefile using -D...

@scargill
Copy link
Author

scargill commented Mar 3, 2019

Eriksl - you have me confused - up to now I've ben working on the asumption that this was required and nothing else. Clealry below is not QUITE right as it won't compile - the usual "won't fit IRAM" message.

but now you've introduced something called partition_items[] - care to elaborate for non-Linux tpyes like me?

#define SDK_RF_CAL_ADDR 0x3FB000
#define SDK_PHY_DATA_ADDR 0x3FC000
#define SDK_PARAM_ADDR 0x3FD000
#define SPI_FLASH_SIZE_MAP FLASH_SIZE_32M_MAP_1024_1024
#define SDK_PRIV_PARAM_ADDR 0xFC000
#define SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM SYSTEM_PARTITION_CUSTOMER_BEGIN

static const partition_item_t p_table[] = {
{ SYSTEM_PARTITION_BOOTLOADER, 0x0, 0x2000 },
{ SYSTEM_PARTITION_RF_CAL, SDK_RF_CAL_ADDR, 0x1000 },
{ SYSTEM_PARTITION_PHY_DATA, SDK_PHY_DATA_ADDR, 0x1000 },
{ SYSTEM_PARTITION_SYSTEM_PARAMETER, SDK_PARAM_ADDR, 0x3000 },
{ SYSTEM_PARTITION_CUSTOMER_PRIV_PARAM, SDK_PRIV_PARAM_ADDR, 0x1000 },
};

void user_pre_init(void)
{
if(!system_partition_table_regist(p_table, sizeof(p_table)/sizeof(p_table[0]), SPI_FLASH_SIZE_MAP)) {
os_printf("system_partition_table_regist fail\r\n");
while(1);
}
}

@kriegste
Copy link

kriegste commented Mar 3, 2019

That's the name of the array!? In your example it is called p_table.

BTW, have you tried ditching rboot completely?

@scargill
Copy link
Author

scargill commented Mar 3, 2019 via email

@eriksl
Copy link

eriksl commented Mar 13, 2019

BTW I also have a script in the makefile to produce a top-X of IRAM using source files (unfortunately not on function level). I appears, I see now, to only work when link time optimisation is off, otherwise all subsections are renamed to .ltrans.ltrans.o which isn't at all that informative.

Interest?

@eriksl
Copy link

eriksl commented Mar 13, 2019

Probably the fastest way to find these potential functions would be to remove libc from the library list - the build will throw a bunch of errors during the link step which will show you which functions you're using. You can then work out which of those should be replaced or moved.

+1. And best try to keep it that way. I ended up with just 3-4 functions from libc being used. Some of them I could implement myself very easily and some of them I just copied the source from a "light" version, e.g. from BSD. The versions from libc are always huge and bloated (due to a.o. locale and backward compatibility), so they fill up IRAM or IROM (when using libcirom) quite easily. That's a real good place to start optimising IRAM.

I can also see a SoftUart init function that may be able to be moved - there's also an interrupt function that obviously can't.

SoftUART? Something like an emulator? Or a bit-banging software implementation? While there are two hardware UARTs?

I have also created libcirom and libmirom libraries that put everything in irom - for those functions that you have to use from libc, these may help:

I think the same goes for libm ;-) I am using powf because I can't replace it (I need to calculate "something" to the power 1.2, quite hard to emulate). The powf function does not end iup in IRAM so I think Espressif or the esp-open-sdk / crosstool-ng already take care of that?

@eriksl
Copy link

eriksl commented Mar 13, 2019 •
edited
Loading

Talking of strings. Very soon after I started with the ESP8266 I realised there is so much memory (I came from ATmel ATmega with just 2kbytes of RAM), which is nice, but also means it should be used sensibly and, more important, using char arrays as strings will open up the same traps we all know have fell into more than once. So I created a struct called "string_t" which includes the size of the string, the current length of the string and a pointer to the actual storage. This it can hold anything and is not limited to zero-terminated data. And then I made quite a few string manipulating functions, all in IROM of course. It's a bit like the C++ std::string class. That way you can be sure you won't be bitten by buffer overflows, never. As a nice side-effect I had to make my own string-to-numeric and v.v. functions which are very lightweight and do not rely on Espressif code (important to me). So I don't need strto(u)l etc. or str(n)cat etc.

Also I tried using pvPortMalloc and pvPortFree (aka os_malloc and os_free) one time and very quickly reverted it. I got all sorts of random memory corruption, while having been using malloc() and free() on Unix for about 25 years, so I do grasp the concept ;-) So now I just allocate everything as static and that works well. Some large blocks of memory I re-use internally, no need to alloc-free-alloc-free there, with risk on fragmentation.

@davydnorris
Copy link

I think the same goes for libm ;-) I am using powf because I can't replace it (I need to calculate "something" to the power 1.2, quite hard to emulate). The powf function does not end iup in IRAM so I think Espressif or the esp-open-sdk / crosstool-ng already take care of that?

Yeah I have rolled my own fixed point routines for everything I need. You can also do a fixed point version of powf pretty easily, and for your case powf(x,1.2) = x * powf(x, 0.2) and that last term will converge very quickly as a series.

I have completely removed any floating point from my code - all the data from sensors is fixed point so I just work with it as fixed point. Initially I started by reporting the data up to my platform as floating point strings, but now I have adopted the approach used by IIO and have defined each measurement as a channel, and the channel definition describes the size, precision, and divisor for each value. I then send a 1-8 byte value depending on the size, and the platform uses the the device metadata to do the conversion. I can also just send the processed floating point string data if required.

I've also abstracted the sensors from the devices, and have built a bus/driver/device framework similar to IIO/Linux Kernel drivers so that I can quickly swap in boards with more or less sensor chips, or upgraded sensor chips, and the sensors adapt to the new devices automatically. All I do is add new driver code for new devices as needed

@davydnorris
Copy link

Talking of strings. Very soon after I started with the ESP8266 I realised there is so much memory (I came from ATmel ATmega with just 2kbytes of RAM), which is nice, but also means it should be used sensibly and, more important, using char arrays as strings will open up the same traps we all know have fell into more than once.
...Some large blocks of memory I re-use internally, no need to alloc-free-alloc-free there, with risk on fragmentation.

That's very cool. I have #define'd all my strings and I do a lot of string concatenation in the define itself. Then I use the optimised os_printf, and I've done same for os_sprintf as well so all the strings automatically get turned into RO data.

I do use the os_*alloc functions all the time though and I haven't had any problems that weren't of my own creation :-).

One thing I have just recently done is to go through all my functions that are part of any callback chains, and juggled things around so I use as few function local variables as possible (to minimise stack), and release/dispose of as much as I can before calling the callback. I've found this to be a very valuable exercise in helping manage memory and stack

@fvpalha
Copy link

fvpalha commented Mar 14, 2019 •
edited
Loading

Hi.

Please @scargill, do you resolved the problem?

Your code does not use the "partition_item_t".
user_main.c

Thank you.

@eriksl
Copy link

eriksl commented Mar 14, 2019

I think the same goes for libm ;-) I am using powf because I can't replace it (I need to calculate "something" to the power 1.2, quite hard to emulate). The powf function does not end iup in IRAM so I think Espressif or the esp-open-sdk / crosstool-ng already take care of that?

Yeah I have rolled my own fixed point routines for everything I need. You can also do a fixed point version of powf pretty easily, and for your case powf(x,1.2) = x * powf(x, 0.2) and that last term will converge very quickly as a series.

Takes me back to maths classes a long time ago :-)

I don't see the harm in using floating point. On my ATmel ATmega I wouldn't dream of it, takes too much RAM and too much cpu cycles, but the lx106 seems to have no problem with it whatsoever. So I gave in here. Great advantage of floating point is that you can put in very large and very small values, it always fits. Or more important, you don't have to worry about intermediate values overflowing (or underflowing) the end result.

I have completely removed any floating point from my code - all the data from sensors is fixed point so

I opted for a simple text based interface, very easy to debug. It now supports 29 distinct I2C sensors. That's all I can find on eBay ;-) If anyone has suggestions for sensors that have an I2C interface, are affordable and can be hand-soldered (i.e. DIP or break-out) I am interested.

It looks like this (that's one of 12 ESP8266's in/around my house):

eos erik:~/src/esp/esp8266-universal-io-bridge $ espif esp2 isd
+ sensor 0/39@4f: lm75 #3, temperature: [29.24] C
+ sensor 1/14@39: tsl2561 #0, visible light: [8.24] lx
+ sensor 1/18@48: tmp75 #0, temperature: [20.62] C
+ sensor 1/59@5c: am2320/1/2, humidity: [46] %
+ sensor 1/60@5c: am2320/1/2, temperature: [19.60] C
+ sensor 2/17@23: bh1750, visible light: [5.54] lx
+ sensor 2/32@4a: ds1621 #2, temperature: [18.0] C
+ sensor 2/43@44: sht30, humidity: [48] 
+ sensor 2/44@44: sht30, temperature: [20.07] C
+ sensor 2/50@76: bme680, air quality: [100] %
+ sensor 2/51@76: bme680, temperature: [20.25] C
+ sensor 2/52@76: bme680, humidity: [46] %
+ sensor 2/53@76: bme680, air pressure: [1002.27] hPa
+ sensor 3/11@4a: max44009, visible light: [6.88] lx
+ sensor 3/40@60: mpl3115a2, temperature: [17.74] C
+ sensor 3/41@60: mpl3115a2, air pressure: [1001.23] hPa
+ sensor 3/61@27: hih6130, humidity: [48] 
+ sensor 3/62@27: hih6130, temperature: [19.76] C
+ sensor 4/42@5a: ccs811, air quality: [99] %
+ sensor 4/46@40: hdc1080, humidity: [54] 
+ sensor 4/47@40: hdc1080, temperature: [19.64] C
+ sensor 4/54@76: bmp280/bme280, humidity: [0.0] %
+ sensor 4/55@76: bmp280/bme280, air pressure: [1004.07] hPa
+ sensor 4/56@76: bmp280/bme280, temperature: [20.77] C

@fvpalha
Copy link

fvpalha commented Mar 14, 2019

I wanted to see a single code working with rboot in SDK 3.1.0.
Someone have a link to this code?

@eriksl
Copy link

eriksl commented Mar 14, 2019 •
edited
Loading

That's very cool. I have #define'd all my strings and I do a lot of string concatenation in the define itself. Then I use the optimised os_printf, and I've done same for os_sprintf as well so all the strings automatically get turned into RO data.

I do use the os_*alloc functions all the time though and I haven't had any problems that weren't of my own creation :-).

Maybe it was a bug that has been resolved now. Anywyway, if we're this tight on memory, I don't like to waste it with a scheme that will cause fragmentation.

One thing I have just recently done is to go through all my functions that are part of any callback chains, and juggled things around so I use as few function local variables as possible (to minimise stack), and release/dispose of as much as I can before calling the callback. I've found this to be a very valuable exercise in helping manage memory and stack

I don't think you need to be that careful using the stack. I've tried and tried to get information about the stack from Espressif, but no answer. So I made my own stack painting mechanism and found out that there is no less than 5 kb (!) stack space. Of that 5 kb, my code uses 2500 bytes at max, and I am not using stack sparingly (exactly to prevent having to use os_malloc/os_free).

I absolutely loathe #define so you won't them much in my code. Almost all strings go into flash automatically using this construct, which looks somewhat ugly but makes great code in the program itself:

#define string_format(dst, fmt, ...) \
do { \
    static roflash const char fmt_flash[] = fmt; \
    string_format_flash_ptr(dst, fmt_flash, ## __VA_ARGS__); \
} while(0)

And then you can simply do string_format(string, "text")) with the text ending up in flash. The string_format_flash_ptr is a function that takes care of the actual formatting, taking care of alignment issues by copying the string, when used, to DRAM and reference it from there.

Last summer someone said that os_printf (which now points to os_printf_plus or something like that) could handle strings from flash. I tried it (ets_vsnprintf, in my case) and immediately got an alignment exception on the first try.

So I stayed with my own "solution", copying the format string to DRAM and then calling ets_vsnprintf to do the real work. The ets_vsnprintf function is pretty complete but sadly does not understand floating point numbers.

@eriksl
Copy link

eriksl commented Mar 14, 2019

I wanted to see a single code working with rboot in SDK 3.1.0.
Someone have a link to this code?

Yes, for example my "esp8266-universial-io-bridge", here: https://github.com/eriksl/esp8266-universal-io-bridge

rboot combined with NONOS SDK git HEAD.

It's really simple if you follow the advice I gave here and in other threads as well.

@fvpalha
Copy link

fvpalha commented Mar 14, 2019

Thank you @eriksl.

@davydnorris
Copy link

Last summer someone said that os_printf (which now points to os_printf_plus or something like that) could handle strings from flash. I tried it (ets_vsnprintf, in my case) and immediately got an alignment exception on the first try.

So I stayed with my own "solution", copying the format string to DRAM and then calling ets_vsnprintf to do the real work. The ets_vsnprintf function is pretty complete but sadly does not understand floating point numbers.

That's very weird because I set #define USE_OPTIMIZE_PRINTF in my code, which turns all os_printf strings into flash and I have had no problems

@eriksl
Copy link

eriksl commented Mar 15, 2019

Yeah, weird. But no big deal, because I have my own string infrastructure anyway, I don't need to use it. Almost all of my strings are already in flash. And as said, I don't trust code from Espressif anyway. Most of it is closed and whatever is not closed, makes me shiver. See for example the I2C code.

Next project is trying to ditch the espconnect code and call lwip directly. That doesn't seem exactly that easy though, documentation about the lwip api is very sparse.

@eriksl
Copy link

eriksl commented Mar 19, 2019

I have not (yet) been agreed to add a subforum for "us" on esp8266.com, let's wait a while and see.

In the meantime I have a few questions about LWIP that may be interesting to others as well. IIRC @davydnorris has become quite knowledgeable about the topic, maybe others too.

Can you please visit https://www.esp8266.com/viewtopic.php?f=6&t=19428 and check my questions?

@d-a-v
Copy link

d-a-v commented Mar 19, 2019

I have a few questions about LWIP

#123 (comment)

@eriksl
Copy link

eriksl commented Mar 19, 2019

That's using espconn emulation. I just finished ditching epsconn :-)

@d-a-v
Copy link

d-a-v commented Mar 19, 2019

That's using espconn emulation. I just finished ditching epsconn :-)

espconn uses lwIP(v1, v2), not the opposite way :)

@eriksl
Copy link

eriksl commented Mar 19, 2019

Yes I know... You're referring (afaik) to a project where lwip is replaced and an espconn emulation layer is added to be able to keep using espconn in the application. I don't need that, I only want a newer lwip and compile it myself.

@d-a-v
Copy link

d-a-v commented Mar 19, 2019

In that project, full lwIP is replaced by the newest available version. You have to compile it yourself.

espconn (which is probably an espressif version of lwIP's netconn) has just been ported to it for those who need it.

@eriksl
Copy link

eriksl commented Mar 20, 2019

I'll have a look. It's not a trivial task to add an external LWIP version, especially another version, because of the Espressif hooks in it for the ESP8266 wlan interface. They didn't always follow the standard low-level LWIP device interface, it seems. So that's why I am a bit careful.

The espconn code is, as far as I'm am concerned, a misguided attempt to cover up that they we're using lwip. It came out very soon and after that, Espressif gave up the secrecy and also most of the support for this interface.

@fvpalha
Copy link

fvpalha commented Mar 20, 2019

Please @eriksl, can you help me?

I changed the main.c, but does not work yet.

https://github.com/fvpalha/test_rBoot_NonOS_SDK3.1.0

@davydnorris
Copy link

I have not (yet) been agreed to add a subforum for "us" on esp8266.com, let's wait a while and see.

In the meantime I have a few questions about LWIP that may be interesting to others as well. IIRC @davydnorris has become quite knowledgeable about the topic, maybe others too.

Can you please visit https://www.esp8266.com/viewtopic.php?f=6&t=19428 and check my questions?

I am so totally not an expert on lwIP! @d-a-v is the genius here big time, not me. But I am very interested in using the latest lwIP and it's in my list of things to do after I push my release.

Right now I just want to be able to change or add to DNS routines to allow me to get the TTL of a query as part of the result. I've implemented a DNS cache that survives deep sleep by using RTC storage and I'm just guessing at the TTL for my entries, which can be dangerous, especially for Cloud based back ends where the IP may change a lot.

@d-a-v
Copy link

d-a-v commented Mar 21, 2019 •
edited
Loading

It's not a trivial task to add an external LWIP version, especially another version, because of the Espressif hooks in it for the ESP8266 wlan interface.

Tell me about it!
espressif made a far better job on rtos-sdk though. I guess we can say "lesson learned".

@d-a-v is the genius here big time

Well that makes lots of me, thanks but this isn't true.
lwIP devs are genius. I am more of a lwIP-with-nonos-sdk hacker :)

Work is in progress for esp-open-sdk (and hopefully nonos-sdk).

@eriksl
Copy link

eriksl commented Mar 21, 2019

Hi guys, the "advanced developer" subforum has been honoured!

Can we continue the talk there?

https://www.esp8266.com/viewforum.php?f=166

I have many questions regarding lwip and I don't think it takes an expert to answer some of them, at least I already very curious about all of your experiences, either succesful or unsuccessful.

@eriksl
Copy link

eriksl commented Mar 21, 2019

Please @eriksl, can you help me?

I changed the main.c, but does not work yet.

https://github.com/fvpalha/test_rBoot_NonOS_SDK3.1.0

What does the UART output say?

@fvpalha
Copy link

fvpalha commented Mar 21, 2019

Please @eriksl, can you help me?
I changed the main.c, but does not work yet.
https://github.com/fvpalha/test_rBoot_NonOS_SDK3.1.0

What does the UART output say?

File main.c

#define FLASH_SIZE_SDK			FLASH_SIZE_32M_MAP_512_512
#define SPI_FLASH_SIZE_MAP		FLASH_SIZE_32M_MAP_512_512
#define RFCAL_OFFSET			0xfb000
#define RFCAL_SIZE			0x1000
#define PHYDATA_OFFSET			0x1fc000
#define PHYDATA_SIZE			0x1000
#define SYSTEM_CONFIG_OFFSET		0x3FD000
#define SYSTEM_CONFIG_SIZE		0x3000
#define USER_CONFIG_OFFSET		0xfa000
#define USER_CONFIG_SIZE		0x1000
#define OFFSET_OTA_BOOT			0x000000
#define SIZE_OTA_BOOT			0x1000
#define OFFSET_OTA_RBOOT_CFG		0x1000
#define SIZE_OTA_RBOOT_CFG		0x1000
#define OFFSET_OTA_IMG_0		0x002000
#define OFFSET_OTA_IMG_1		0x202000
#define SIZE_OTA_IMG			0xF0000

static const partition_item_t p_table[] = {
		{	SYSTEM_PARTITION_RF_CAL, 		RFCAL_OFFSET,			RFCAL_SIZE,			},
		{	SYSTEM_PARTITION_PHY_DATA,		PHYDATA_OFFSET,			PHYDATA_SIZE,			},
		{	SYSTEM_PARTITION_SYSTEM_PARAMETER,	SYSTEM_CONFIG_OFFSET,		SYSTEM_CONFIG_SIZE,		},
		{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 0,	USER_CONFIG_OFFSET,		USER_CONFIG_SIZE,		},
		{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 1,	OFFSET_OTA_BOOT,		SIZE_OTA_BOOT,			},
		{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 2,	OFFSET_OTA_RBOOT_CFG,		SIZE_OTA_RBOOT_CFG,		},
		{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 3,	OFFSET_OTA_IMG_0,		SIZE_OTA_IMG,			},
		{	SYSTEM_PARTITION_CUSTOMER_BEGIN + 4,	OFFSET_OTA_IMG_1,		SIZE_OTA_IMG,			},
};

void user_pre_init(void);
void user_pre_init(void)
{
    if(!system_partition_table_regist(p_table, sizeof(p_table) / sizeof(*p_table), FLASH_SIZE_SDK)) {
        os_printf("system_partition_table_regist fail\r\n");
        while(1);
    }
}

Console: picocom 74880 /dev/ttyUSB0

ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 856, room 16 
tail 8
chksum 0xfa
load 0x3ffe8000, len 236, room 0 
tail 12
chksum 0x76
csum 0x76
mismatch map 4,spi_size_map 6
boot not set
ota1 not set
ota2 not set
system_partition_table_regist fail

@kriegste
Copy link

Isn't FLASH_SIZE_32M_MAP_512_512 a bit small for 0xF0000 images?
Try FLASH_SIZE_32M_MAP_1024_1024 (this is actually what you seem to use with esptool during flashing).

@fvpalha
Copy link

fvpalha commented Mar 21, 2019

I changed the Makefile (6 to 4).

# SPI_SIZE_MAP
# 0 : 512 KB (256 KB + 256 KB)
# 1 : 256 KB
# 2 : 1024 KB (512 KB + 512 KB)
# 3 : 2048 KB (512 KB + 512 KB)
# 4 : 4096 KB (512 KB + 512 KB)
# 5 : 2048 KB (1024 KB + 1024 KB)
# 6 : 4096 KB (1024 KB + 1024 KB)
SPI_SIZE_MAP ?= 4

and:

ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 856, room 16 
tail 8
chksum 0xfa
load 0x3ffe8000, len 236, room 0 
tail 12
chksum 0x76
csum 0x76
boot not set
ota1 not set
ota2 not set
rf cal sector: 251
freq trace enable 0
rf[112] : 00
rf[113

Now, the ESP is working.

@fvpalha
Copy link

fvpalha commented Mar 21, 2019

Isn't FLASH_SIZE_32M_MAP_512_512 a bit small for 0xF0000 images?
Try FLASH_SIZE_32M_MAP_1024_1024 (this is actually what you seem to use with esptool during flashing).

main.c

#define FLASH_SIZE_SDK FLASH_SIZE_32M_MAP_1024_1024

Makefile

# SPI_SIZE_MAP
# 0 : 512 KB (256 KB + 256 KB)
# 1 : 256 KB
# 2 : 1024 KB (512 KB + 512 KB)
# 3 : 2048 KB (512 KB + 512 KB)
# 4 : 4096 KB (512 KB + 512 KB)
# 5 : 2048 KB (1024 KB + 1024 KB)
# 6 : 4096 KB (1024 KB + 1024 KB)
SPI_SIZE_MAP ?= 6

Console: picocom 74880 /dev/ttyUSB0

ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x40100000, len 856, room 16 
tail 8
chksum 0xfa
load 0x3ffe8000, len 236, room 0 
tail 12
chksum 0x76
csum 0x76
boot not set
ota1 not set
ota2 not set
rf cal sector: 251
freq trace enable 0
rf[112] : 00
rf[113

Everything is working.

Thank you @kriegste.

@fvpalha
Copy link

fvpalha commented Mar 21, 2019

https://github.com/fvpalha/test_rBoot_NonOS_SDK3.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@marcelstoer @eriksl @scargill @fvpalha @d-a-v @someburner @davydnorris @kriegste