Request -> Response 2.8ms time requirement and storing parameters in flash

[alvangee]

Hello!

Everything seems to work fine, except for one issue.

When running OLA RDM Tests, my device fails at 6 of total 407 tests.
All these tests involve changing parameters: DMX start address, Factory Defaults, Device Label.



I am implementing RDM Responder using Atmel xMega MCU. For nonvolatile storage of PIDs value (including DMX Start Address, Device Label, Device Hours, Device Power Cycles and some Manufacturer parameters) are stored in controller's flash memory (to be exact, in Apptable section of flash memory).

Here's what I do when SET command received for those PIDs:
1) Send ACK for SET_COMMAND
2) Turn OFF all interrupts
3) Update parameter value in MCU's flash memory by performing atomic
Erase-Write operation
4) Turn ON all interrupts

The issue is in timings. Each change of any of those parameter value by SET_COMMAND requires reading of page of flash memory, erasing that page of memory (this takes ~4 ms), then writing of page of memory (takes another ~4 ms).

Although, the E1.20 states that maximum time between any Controller:Request and Responder:Response is 2.8 ms. Obviously, it is impossible for my device to comply with this requirement running OLA RDM Tests, because right after SET_COMMAND for any of said PIDs there is GET_COMMAND for that exact PID and for the time of this GET_COMMAND MCU is busy with Erase-Write Flash Memory Page operation, which takes ~8 ms.

Thus, when any command/request follows SET_COMMAND for any of flash-memory-stored parameter earlier than ~10 ms, I fail at responding in required 2.8 ms.

Waiting for no bus activity for 1.25 second and only then perform erase-write flash memory operation doesn't seem as a solution - still no garantees that Controller won't send command or request at any time after 1.25 second.

Maybe I am doing it all wrong and there is some industry approved way of storing parameters values in compliance with 2.8 ms requirement?

How do you usually store parameters in your devices?

Best regards, Alexey Ivakin.

[sblair]

I would consider it to be a recoverable error in that the controller will have not gotten a response for the GET so it can always treat it as lost and send another one.

Instead of sending an ACK for the SET you can always send an ACK_TIMER. That tells the controller that it can't do an immediate ACK. That doesn't stop the controller from sending a different request before the ACK_TIMER expires but should reduce the odds.

The other, better but more painful, approach would be to try re-working your flash erase/write cycle to spread it out so you can still handle incoming requests.

[ericthegeek]

Can you store the values in EEPROM instead of flash? I don't know the xMega family, but skimming the datasheet it looks like most of them have both Flash and EEPROM.

Many microcontrollers that have the ability to self-write their flash memory will stall the CPU core during a flash erase/write. But you can topically start writing to the EEPROM and then keep executing code while the EEPROM write is in progress. With this, you can save the new data in RAM, ACK it to the RDM controller, and then start writing it to EEPROM as a background task. This leaves the CPU free to respond to any new RDM traffic that occurs during the write.

Another option is that some MCUs have multiple flash banks that allow you to execute from one bank white you erase/write the other.


RDM is not a reliable protocol: Any device can drop any packet at any time for any reason (or no reason). Thus, as Scott said, you *could* ACK the request and then just ignore any DMX traffic until you've finished the 8ms write operation. You could also ACK_TIMER and do the same. But best practices call for responders to be able to receive and process all packets on the wire as long as they are sent with valid timing. You should only drop packets as a last resort.


Finally, don't forget about broadcast and vendorcast packets. Responders have 2.0ms to respond to unicast requests (E1.20 table 3-4), but broadcast packets can be sent back-to-back with only 176us between them. So you can't even depend on having 2.0ms to process the SET. A broadcast "SET FACTORY_DEFAULT" is perfectly legitimate, and a well-designed responder will be able to handle a new packet that comes 176us later even if it has to do many milliseconds of work in the background to reset everything to default.

[alvangee]

Thanks for the answer, Scott.

I'll consider sending ACK_TIMER. There's no support of Queued Messages in the firmware at this moment, looks like it's time to add it.

Unfortunately, flash erase/write cycle can't be modified - it is atomic operation which should be done uninterrupted and works only on pages of 256 bytes.

It would be possible to separate flash operation if it was esternal flash chip, but we need to use internal MCU flash memory for storage.

On the other hand, for me it looks just not right to send an ACK to any "change important parameter" command before this parameter was securely stored in nonvolatile memory of any sort. There is a possibility that after sending ACK the power of device was shut down and after restart there will be undesirebale parameter value.
Was this issue ever been discussed?

[alvangee]

Originally Posted by ericthegeek

Can you store the values in EEPROM instead of flash?

Hell, my supervisor doesn't trust EEPROM in xMegas. He says there's been some incidents where valuable data (for example, start DMX address) in previous projects were lost, so internal flash is my only option.

As I see it, there's no way to make a well-designed responder with this configuration, I need to use external memory chip for this. With internal memory there's always will be some 8 ms cycles when responder will drop any messages while it refreshes values in flash.

It's even worse with SET_FACTORY_DEFAULT, when several parameters in different flash memory pages are adjusted which takes 8ms for each of this pages to erase and write.

Not sure about executing from one bank while writing to another. I'll check if this a possible operation on xMega. At least I am storing my parameters in special Apptable section of flash, not just somewhere in program section of it.

Thanks for the replies!

[sblair]

Originally Posted by alvangee

On the other hand, for me it looks just not right to send an ACK to any "change important parameter" command before this parameter was securely stored in nonvolatile memory of any sort. There is a possibility that after sending ACK the power of device was shut down and after restart there will be undesirebale parameter value.
Was this issue ever been discussed?

Yes, its been discussed a number of times. It really is up to the implementer. In some cases where the write operation is considered to be reliable enough a regular ACK will be used. Others that are more concerned with the outcome of the write operation will use the ACK_TIMER so they can verify the write occurred successfully. In the end it's really your choice.

[ericthegeek]

Originally Posted by alvangee

Hell, my supervisor doesn't trust EEPROM in xMegas. He says there's been some incidents where valuable data (for example, start DMX address) in previous projects were lost, so internal flash is my only option.

I've dealt with a lot of NVM (Flash and EEPROM) corruption issues over the years. It's nearly always an implementation problem, I'm sure the Atmel part's EEPROM is fine. Often times when powering up or down, the software will start to execute at a random address. Make sure you have a reset supervisor that prevents the CPU from running unless the power is stable and well within the CPU's operating range.

Also, never write code that looks like this:
write_enable_nvm()
unprotect_nvm()
write_nvm(*addr,data)

Instead,
write_enable_nvm()
(...require some external event...)
unprotect_nvm()
write_nvm(*addr,data)

This way, if the software goes haywire and starts executing at a random address, it can't accidentally overwrite whatever *addr happens to be pointing to at that moment. Also, make sure you disable writes ASAP if the required external event doesn't happen within the expected time period.

This is true whether it's Flash or EEPROM. If your firmware (or bootloader) has code that is capable of writing NVM, you need to make darn sure that code never executes unless it's supposed to.

EEPROM really is the right solution here. If he or she doesn't trust the internal EEPROM, external EEPROMs are cheap.

Originally Posted by alvangee

As I see it, there's no way to make a well-designed responder with this configuration, I need to use external memory chip for this. With internal memory there's always will be some 8 ms cycles when responder will drop any messages while it refreshes values in flash.

Unfortunately you are correct. A well behaved RDM responder really shouldn't go out to lunch for multiple milliseconds. You can probably get away with it, but it's not good practice.

Originally Posted by alvangee

On the other hand, for me it looks just not right to send an ACK to any "change important parameter" command before this parameter was securely stored in nonvolatile memory of any sort.

Some people feel this way, that you must have stored the new data in Non-volatile memory before you ACK the request. Personally, I don't feel this is necessary. RDM is not a database where you need atomic operations and absolute certainty that the data is written. If someone sets a DMX address, and the rig loses power a few milliseconds later, they won't be shocked and appalled if it has the old DMX address when power comes back on. My personal opinion is that if you've validated the new setting, and have scheduled the write to Flash/EEPROM, it's acceptable to ACK the request.

I also feel this way because ACK_TIMER/QUEUED_MESSAGE isn't implemented perfectly in all RDM controllers. Some controllers don't use Queued Messages at all. Based on this, I believe that overall using ACK_TIMER will cause you more problems than just using ACK and scheduling the write. You can't schedule the write and wait a week to commit it to memory. but a few dozen milliseconds won't hurt anything.