iGate receiver removes duplicate packets

[hessu]

Hi,

It would seem to me that the direwolf igate, in igate.c rx_to_ig_allow(), removes duplicate packets based on a CRC check.

It would be better for the APRS-IS servers, if they would get to see all the received duplicate packets. Having the duplicates available there gives the servers more packet path information, and makes it possible to generate better network maps and such, based on aggregated duplicate packets on the server side.

They also have a more exact duplicate matching algorithm, which uses a hash lookup to find the duplicates. aprsc also compares the actual packet data, not just the checksum used to key the hash. This removes the problem of accidental drops due to hash collisions.

Having a fewer number of duplicate filtering algorithms (just one in the APRS-IS servers) also gives more consistent network behaviour.

Could you consider removing the duplicate checking from the RX igate code? For transmitting and digipeating, it is of course necessary.

Thank you!

Hessu, OH7LZB (of aprsc, aprs.fi)

[hessu]

Hi,

I just wrote a bit more text here about the hash collision problem. direwolf seems to use a 16-bit CRC check alone for duplicate packet detection. I've seen a few other igates doing the exact same thing, ArcticTracker at https://github.com/ohanssen/ArcticTracker/blob/dd6d20ca0f3e3d3e1ec652c6e9c3980095692bdb/igate.c#L267 is one example.

https://github.com/hessu/aprsc/blob/master/doc/IGATE-HINTS.md#igates-dropping-duplicate-packets-unnecessarily

Given 100 random packets, there is a 7.3% probability that two packets will have the same checksum. Given 300 random packets, 49.6% probability that two different packets have the same CRC. If an igate is running for weeks and months, there will be some extra packets dropped if CRC16 is the only check.

[CInsights]

Hi,
FYI the local CRC de-dupe algorithm is implemented using a limit of 30 recent packets and an expiry time of 60 seconds.
Hence the probability calculations need to factor in arrival rate, finite limit of packets compared and time limit expiration.

For the code see igate.c around line 1964 (correct as of current dev branch).

[hessu]

Hello,

Nonetheless, the receiving igate does not need to have a duplicate filter, it's quite simply not necessary.

[CInsights]

Something for the author to consider.
Perhaps you could write up a modification and propose it as a pull request?
Maybe make it an optional setting through the command line or config file directive?

[wb2osz]

Disabled duplicate checking for the IGate RX>IS direction.
b81de60

[hessu]

Excellent, thank you!