There is no way that charcoal filter isn't causing an airflow restriction, just physics.
Now, if you are saying that restriction is taken into account by the air flow mixture and tuning of the engine, absolutely that makes sense.
That being said, my main reason to remove it is I'm afraid over time it will slowly disintegrate, pieces getting inside and damaging the engine. Besides, the charcoal filtration was depleted years ago, the fact it isn't replaceable is a joke if it was meant for emissions.
The factory Mopar intake removes it anyways, it doesn't need to be there even for CARB/EPA reasons. I say remove it before it becomes a problem. Instead of removing in place, highly recommend you remove the top of your box (disconnect from intake tube), cut out the filter, clean any potential debris before reinstalling.