UF2 considerations

[microbit-carlos]

I'm looking at adding UF2 on top of the Universal Hex out-of-order blocks implementation:

• Process out-of-order vfs sectors for Universal Hex in block format #1092

I've rebased this branch, which has a basic UF2 implementation already working:

• develop...mbrossard:DAPLink:feature/uf2

There are a few things we need to consider before finishing the implementation.
Everything here is up for discussion, this is just my initial proposal.

Won't Implement

Some things that I'll l likely not implement at this point:

• File containers
  • This requires knowledge in DAPLink about the target filesystem, so it's generic enough for all boards
  • Any particular board that might want to implement this could probably add hook into write_uf2() and include it in their port
• CURRENT.TXT
  • No special reason, simply trying to reduce scope for now
  • If implemented it should probably be with a compile-time flag selectable per port
• MD5 checksum
  • target_flash_program_page() can already verify data written if automation is enabled
  • But this is more useful to avoid writing data to flash (reduces flashing time). So, this is only useful when page-by-page erase and programming is enabled, as otherwise it does a full chip erase when it receives the first valid data block.

Could implement

Some things that are not implemented yet but can consider adding.

• Tracking that all individual blocks have been received
  • This requires a buffer with a bit per block, so for a device with 256 KB and 256 bytes per block, it'd need 1 KB of RAM. Unsure if this is viable for all DAPLink ports
    • Alternatively this feature could be turned off by default, and enabled via compile-time flag
  • UF2 files with multiple targets could also require an arbitrary number of flag buffers, although we could only track the blocks relevant for the current target
    • Clarification about blockNo and numBlocks microsoft/uf2#91
  • vfs_manager tracks the UF2 file size, so and if fewer blocks than expected arrive it will throw an error
    • However, this does not protect from the OS sending the same block more than once (perhaps it creates a temporary file, or metadata file)

Open questions

1. Should UF2.TXT file be created in the MSC drive?
   • If implemented we might want to add it as a compile-time flag as well
2. UF2 FamilyID corresponds to a generic MCU family or specific MCU variants, but it doesn't quite match the Mbed/DAPLink Board ID
   • 2.1) We could try to allocate a range of FamilyID values where the top 2 most significant bytes represent "this is DAPLink family" and the 2 least significant bytes match the Board ID
   • 2.2) We could use extension tags. The existing "device type identifier" could work, but we could create a DAPLink specific one to include the Board ID.
   • 2.3) Should we also expand target_family_descriptor to include the UF2 family IDs for supported families?

[microbit-carlos]

2. UF2 FamilyID corresponds to an MCU family or specific MCU variants, but it doesn't quite match the Mbed/DAPLink Board ID
   • 2.1) We could try to allocate a range of FamilyID values where the top 2 most significant bytes represent "this is DAPLink family" and the 2 least significant bytes match the Board ID
   • 2.2) We could use extension tags. The already defined "device type identifier" might already work, or create a DAPLink specific one to include the Board ID.

@mmoskal I'm currently looking into adding UF2 into DAPLink, and I was wondering if you'd consider option 2.1 listed above, and reserve something like the 0xDA91xxxx range for DAPLink board IDs?
A list of board IDs can be found in: https://github.com/ARMmbed/DAPLink/blob/main/test/info.py#L142

[mathias-arm]

I agree with your "Won't Implement" items and I don't think that we should work on the "Could implement" item until it is an actual requirement.

On the Open Questions:

1. I would consider that a low priority item. I don't have good visibility on the practical added value to users.
2. I have some concerns about aligning DAPLink/Mbed Board IDs and UF2 Family IDs. I prefer the idea in your 2.3 point. We should extend HIC and Targets to expose their UF2 Family IDs, so that DAPLink can support UF2 for target flashing but also its updates (bootloader and interface).

[microbit-carlos]

I agree with your "Won't Implement" items and I don't think that we should work on the "Could implement" item until it is an actual requirement.

Sounds good 👍

On the Open Questions:

1. I would consider that a low priority item. I don't have good visibility on the practical added value to users.

Perhaps tools or editors/IDEs that scan for the UF2.TXT file to detect UF2 devices. Not sure if DAPLink users would benefit from this, but mostly I'm not familiar with the tools that might check it.

2. I have some concerns about aligning DAPLink/Mbed Board IDs and UF2 Family IDs. I prefer the idea in your 2.3 point. We should extend HIC and Targets to expose their UF2 Family IDs, so that DAPLink can support UF2 for target flashing but also its updates (bootloader and interface).

Extending something like target_family_descriptor to include the UF2 Family ID would still require the DAPLink Board ID to be embedded somehow, no?
Or are you suggestion UF2 files to not contain the DAPLink board ID?
The same target MCU could have different configurations resulting in a different Board ID, for example micro:bit v1.3 is Board ID 9900 and v1.5 is 9901, both having the same Interface and Target MCUs (the difference was the on-board motion sensors). In this specific case the same hex file would work on both boards, but the idea was to have a way to tell them apart future revisions in case we needed to create separate builds.

One specific advantage of option 2.1 (a range of DAPLink Family IDs that with the Board ID as the 2 least significant bytes) is that if we had a single UF2 file with separate flash data for two different Board IDs with the same Target MCU, this would result in using 2 different Family IDs. The advantage here that each BoardID/FamilyID would have its own range of blocks, so, FamilyID 0xDA91_0001 can have 20 blocks 0 to 20, and FamilyID 0xDA91_0002 can have 15 blocks 0 to 15.
If instead both boards used the same FamilyID (and the Board ID included only via "extension tag"), then we would end up with 35 blocks 0 to 35, and multiple blocks would target the same address, for example block 0 and block 21 would both target address 0x0 (block 0 for Board ID 0001 and block 21 for Board ID 0002).

[mmoskal]

@microbit-carlos I would suggest doing both 2.1 and 2.3 in the bootloader, that is I'm happy with 0xDA91_xxxx range but the bootloader should accept both the DA91 and the UF2 family ID for MCU family as defined in uf2 repo. If it accepts more than one ID, it should record the ID it is accepting when it first sees them and stick to that one until the end of flashing.

This way, for micro:bit you could use the ID for NRF52833, but if you need to you can also use 0xDA919901. You should probably also define a family ID for micro:bit V2 in uf2 repo and accept all 3 in bootloader.

The INFO_UF2.TXT file is used for scanning which drives support UF2. MakeCode command line does this, not sure about others.

[elfmimi]

I remember I created this PR draft.

#935

I made Uf2 to ack just like a variant of HEX in binary form.

Anyway, I'm here to test new code. So let's explore the awesome features UF2 is going to provide.

[microbit-carlos]

Thanks everyone!

I would suggest doing both 2.1 and 2.3 in the bootloader, that is I'm happy with 0xDA91_xxxx range

Yeah, I agree this sounds like the best option.

but the bootloader should accept both the DA91 and the UF2 family ID for MCU family as defined in uf2 repo.

Just to clarify, in this case with "bootloader" you mean the DAPLink Interface that is responsible for flashing the target MCU, or do you mean the DAPLink bootloader responsible for updating the DAPLink Interface?

If it accepts more than one ID, it should record the ID it is accepting when it first sees them and stick to that one until the end of flashing.

I hadn't done this in my initial implementation (mostly to keep it simple). I didn't noticed this in other UF2 bootloaders, but I guess the majority would be design to accept a single FamilyID (even though there is a FamilyID for nRF52833 and another for nRF5 generic, implementations are likely only including one).
My initial thought was to consider UF2 files with multiple compatible family IDs to be invalid, but it is true that without this the target will end up with garbage in flash.

I assume most UF2 bootloaders would also struggle with a UF2 file that contains blocks with a valid Family ID and blocks without a Family ID? Or do they protect against that?

This way, for micro:bit you could use the ID for NRF52833, but if you need to you can also use 0xDA919901. You should probably also define a family ID for micro:bit V2 in uf2 repo and accept all 3 in bootloader.

We have a "generic micro:bit V2" Board ID, and then a separate one as well for V2.00, V.21 and V2.2. Might be simpler to use the 0xDA91_xxxx + "generic micro:bit V2" (0xDA91_9903) as the "micro:bit V2 Family ID" in the UF2 json file?
Otherwise would there be a reason not use the nRF52833 Family ID?

The other thing to consider is that in cases like the micro:bit V2.2x where the Interface and Target MCUs could both have the same Family ID, as they can both be an nRF52833 (although technically it is treated like a nRF52820 and it could use that FamilyID instead). In this type of situation a DAPLink Interface UF2 file could be accidentally flashed into the target, or the other way around, and being able to avoid that is a useful feature.

One way to approach this could be to reserve 2 ranges of Family IDs, one specifically for the DAPLink Interface and another for the Target.
Alternatively we could use an extension tag to identify bootloader UF2 blocks.

My preference would be to have another range for DAPLink interfaces, and the DAPLink bootloader could only permit that range for updating the Interface (so it would ignore other "compatible" Family IDs).

---

@mathias-arm are you happy with the discussion so far?
In the end the UF2 family ID fits better in the target_cfg_t structure.

---

Thank you @elfmimi! Your work is present in the branch I built on top of:
develop...mbrossard:DAPLink:feature/uf2

I'll push my latest changes before the end of the week, just need to debug an issue timing out when there are multiple family IDs in one file.

[mathias-arm]

I think the UF2 Family ID discussion is interesting, but still confusing.

What I want to understand how:

• the DAPLink interface is going to determine whether the UF2 is an appropriate image for the target (e.g. nRF52833) or is an appropriate update for the DAPLink bootloader.
• the DAPLink bootloader is going to determine whether the UF2 is an appropriate update for the DAPLink interface.

I think the distinction can be done between bootloader and interface based on the addresses.

[microbit-carlos]

I've pushed to my fork the current WIP branch, as it might help illustrate my current approach. Consider this branch ongoin and untested:

• microbit-carlos/DAPLink@ignore-delete-soon...feature/uf2-ooo

The DAPLink Interface will flash a block if:

• There is no Family ID
• The Family ID matches the ID configured in g_board_info.target_cfg->uf2_family_id
  • This is the official UF2 IDs and are optionally configured per target, in this branch I've added the IDs for nRF52 MCUs and a few STM32Fxxx
• The Family ID matches the DAPLink range, where the 2 least significant bytes are the Board ID, i.e. 0xDA910000 | (uint16_t)board_id
  • A different numeric value for this range can be used instead, DA91 was only chosen to look similar DAPL

https://github.com/microbit-carlos/DAPLink/blob/97de0d61bb7114cabfb2d5ec772f1166f07e614d/source/daplink/drag-n-drop/file_stream.c#L525-L530

These checks can be overwritten in each target via g_board_info.uf2_block_compatible() and I've done so for micro:bit V2, where it also needs to accept an additional Board ID for the generic V2 (9903):
https://github.com/microbit-carlos/DAPLink/blob/97de0d61bb7114cabfb2d5ec772f1166f07e614d/source/board/microbitv2/microbitv2.c#L479-L491

• the DAPLink interface is going to determine whether the UF2 is an appropriate image for the target (e.g. nRF52833) or is an appropriate update for the DAPLink bootloader.
• the DAPLink bootloader is going to determine whether the UF2 is an appropriate update for the DAPLink interface.

My suggestion for this would (which is not implemented in the branch) be to have a second "DAPLink range" of Family IDs, but only to identify the DAPLink Interfaces (by the DAPLink bootloader).
So essentially we end up with 2 ranges, 1 for "DAPLink Interfaces" and 1 for "DAPLink targets".
Also, the DAPLink bootloader should only accept UF2 blocks with the "DAPLink Interface" Family ID, so the target_cfg structs for the bootloaders should not contain the uf2_family_id field (or the bootloader can simply ignore them).

So this way:

• DAPLink interface can reject files/blocks intended for the bootloader to update the DAPLink interface
• DAPLink bootloader can ensure it only flashes DAPLink Interface files/blocks
  • So a target UF2 file is rejected by the bootloader

Alternatively we could achieve a similar end result using an "extension tag" for DAPLink Interface data, for the bootloader to identify them and flash them, and the Interface to reject them.

I think the distinction can be done between bootloader and interface based on the addresses.

Yes, but without the extra family ID or extension tag, the bootloader might not be able to tell the difference between Interface and Target data. Technically the interface will start at address 0x8000, but since the blocks could arrive in any order we might not know where the first block address starts until half way through the file.

[mmoskal]

but the bootloader should accept both the DA91 and the UF2 family ID for MCU family as defined in uf2 repo.

Just to clarify, in this case with "bootloader" you mean the DAPLink Interface that is responsible for flashing the target MCU, or do you mean the DAPLink bootloader responsible for updating the DAPLink Interface?

Forgot about that one! Also forgot the DAPLink interface runs all the time ;) I was talking about it.

If it accepts more than one ID, it should record the ID it is accepting when it first sees them and stick to that one until the end of flashing.

I hadn't done this in my initial implementation (mostly to keep it simple). I didn't noticed this in other UF2 bootloaders, but I guess the majority would be design to accept a single FamilyID (even though there is a FamilyID for nRF52833 and another for nRF5 generic, implementations are likely only including one). My initial thought was to consider UF2 files with multiple compatible family IDs to be invalid, but it is true that without this the target will end up with garbage in flash.

I assume most UF2 bootloaders would also struggle with a UF2 file that contains blocks with a valid Family ID and blocks without a Family ID? Or do they protect against that?

Right, I'm pretty sure most bootloaders will fail if you give them UF2 file with multiple "options" (either multiple matching familyID or no family ID). That is unless the blocks are in order, in which case they will reboot after the first one is done.

If it's not too much bother, it would be nice to support this family-latching (and treat no family as family 0 or something), for better user experience.

This way, for micro:bit you could use the ID for NRF52833, but if you need to you can also use 0xDA919901. You should probably also define a family ID for micro:bit V2 in uf2 repo and accept all 3 in bootloader.

We have a "generic micro:bit V2" Board ID, and then a separate one as well for V2.00, V.21 and V2.2. Might be simpler to use the 0xDA91_xxxx + "generic micro:bit V2" (0xDA91_9903) as the "micro:bit V2 Family ID" in the UF2 json file? Otherwise would there be a reason not use the nRF52833 Family ID?

0xDA91_9903 sounds good!

The other thing to consider is that in cases like the micro:bit V2.2x where the Interface and Target MCUs could both have the same Family ID, as they can both be an nRF52833 (although technically it is treated like a nRF52820 and it could use that FamilyID instead). In this type of situation a DAPLink Interface UF2 file could be accidentally flashed into the target, or the other way around, and being able to avoid that is a useful feature.

I'm starting to feel this familyID is more like productID, with the "fallback" familyID being the whole MCU family. This is not so important for pure dev boards used by pro-devs but much more important in cases like micro:bit. So I think for the interface UF2 images I would invent new unrelated IDs. You don't want people flashing random NRF52833 images on your interface chip.

One way to approach this could be to reserve 2 ranges of Family IDs, one specifically for the DAPLink Interface and another for the Target. Alternatively we could use an extension tag to identify bootloader UF2 blocks.

My preference would be to have another range for DAPLink interfaces, and the DAPLink bootloader could only permit that range for updating the Interface (so it would ignore other "compatible" Family IDs).

Ranges are good. But does DAPLink have different 99xx numbers for micro:bits with different interface chips?

[microbit-carlos]

If it's not too much bother, it would be nice to support this family-latching (and treat no family as family 0 or something), for better user experience.

Yeah, I think that should be fine.
Should the UF2 spec indicate that Family ID 0x0000_0000 is reserve for no use?

Looking into the order of checks, to keep things simple I'm thinking UF2 blocks without any Family ID could always be flashed.
I'd probably considered a UF2 file to be invalid if only some blocks containing a Family ID.

0xDA91_9903 sounds good!

Great, at some point I'll send a PR with the micro:bit V1.x and V2.x IDs.

So I think for the interface UF2 images I would invent new unrelated IDs. You don't want people flashing random NRF52833 images on your interface chip.
...
Ranges are good.

Great, so I was thinking 0xDA90_xxxx range for DAPLin Interface images?

But does DAPLink have different 99xx numbers for micro:bits with different interface chips?

Yes, 9903 is a generic ID for all V2 boards, 9904 is for boards specifically with a KL27z Interface, 9905 for nRF52833 Interface, and 9906 for nRF52820 Interface.
The difference between V2.20 and V2.21 is a different component on the board, so confusingly we might have V2.20 and V2.21 boards with either nRF52 Interface MCU. But we use the same Interface image for both of them.

[mmoskal]

We can definitely reserve family 0!

Looking into the order of checks, to keep things simple I'm thinking UF2 blocks without any Family ID could always be flashed.
I'd probably considered a UF2 file to be invalid if only some blocks containing a Family ID.

Alternatively, you could just not flash blocks with no family ID at all. There are no legacy UF2 files to support in this case.

DA90 is good.

[mathias-arm]

I was looking at the UF2, and I was wondering if we could use extension tags so we would not need more than one Family ID range. No tag will mean the file is for the target. For differentiating between bootloader and interface, I see two solutions: either we define two tags with no value or one tag with two values (0/1?).

[microbit-carlos]

Yeah, if you prefer using a tag that sounds good too. Hopefully the inlined daplink_is_bootloader() compiles out the checks in the interface builds.

either we define two tags with no value or one tag with two values (0/1?).

A tag without value is probably the fastest implementation, as it can just be a loop with an initial uint32_t comparison, and if that fails, only then it checks the tag size to find the next tag.
Realistically if we are not really going to check this tag in the interface we don't even need the second tag, just a single "this is a DAPLink bootloader block", but we could reserve the tag ID just in case.

[mathias-arm]

Ok, tags without values are better.
I was thinking two tags one for interface and one for bootloader:

• interface flashes to target if there is no tag, flashes to bootloader if bootloader tag is found (ignores interface tag).
• bootloader flashes to interface if interface tag is found (ignores if bootloader tag or no tag found).

I guess one could do with one tag and distinguish bootloaders from interface with the addresses, but addresses have shifted in the past so I would prefer something more explicit.

[microbit-carlos]

Ah, I see what you meant now, thanks for the clarification!

I'm a bit worried about the interface updating the bootloader on the fly, in case we don't receive all bootloader blocks for whatever reason. The current method of bundling a bootloader image with the interface, and then the interface on first boot doing a checksum verification and updating the bootloader feels more robust.
That being said, I've never actually tried it, are there any limitations to using this method? Maybe not all ports have enough free space in the interface flash region to fit a bootloader update?