ntpd relies on the underlying operating system to protect it from requests that impersonate reference clocks. Because reference clocks are treated like other peers and stored in the same structure, any packet with a source ip address of a reference clock (127.127.1.1 for example) that reaches the receive() function will match that reference clock's peer record and will be treated as a trusted peer.
Any system that lacks the typical martian packet filtering  which would block these packets is in danger of having its time controlled by an attacker. The read_network_packet() function in ntp_io.c addresses this issue for IPv6 with Bug 2672, because some OS's were known to not block spoofed ::1 addresses.
Some, if not all, reference clock drivers set their origin values to zero. It would be trivial to control time on a system with one of these drivers configured if these spoofed packets were not filtered.
ntpd should filter IPv4 packets with the address of 127.127.x.x for proactive protection for systems that fail to implement martian packet filtering.  https://en.wikipedia.org/wiki/Martian_packet
CVSSv2: 2.6 - (AV:N/AC:H/Au:N/C:N/I:P/A:N) CVSSv3: 3.7 - CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:L/A:N
Modern Operating Systems already mitigate this vulnerability. For instances where the user can't rely on their OS to filter martian packets, they could introduce additional filtering though a firewall.