Interestingly my local inspector said an Ev charger is considered an appliance and MUST be installed into plug so there is a disconnect other than the breaker. So mine is into a plug, 50amp fuse.
The NEC is still evolving on this but as I understand it, level 1 chargers (120V) can be cord & plug. Level 2 chargers (240V, 30-100A) cannot be plug connected. The Rivian Wall Charger is Level 2. I don't believe what you have installed is code compliant but your incpector gets final say.
From a practical perspective I don't think that there is a problem with your setup assuming the correct wire sizes etc. The vehicles are smart enough to stop power flow very quickly so if someone unplugged the NEMA 14-50 plug while you're vehicle was charging, the danger should be minimal. I'd suggest making sure the breaker is AFCI.
Can you post a picture of your charger and outlet? I'm very curious to see it. I'm guessing that SJ cord was used from the wall unit to the plug. The body of the wall unit is plastic and isn't made to support SJ cord (what is mentioned as strain-relieve in the above posts).
A plug is not an acceptable means of disconnect. A disconnect is a switch that can be locked - think of an AC condenser disconnect. Is there also a fuse in your setup along with a breaker?
The portable charger that comes with the vehicle is designed to work with 240V 50A outlet.
How has everything been working? What do you have the charger amperage set at?