You can use local commands without the need of an LLM to parse all the commands.
Hooking an LLM to Home Assistant is a bitch an half because it requires a gigantic model to be smart as you expect it to be, and surpassing Alexa or Google Home in some cases.
I have a full write up about this: https://forum.level1techs.com/t/home-assistant-exploration-of-fully-local-voice-assist
Which Zigbee adapter are you using? Which Zigbee implementation?
I’m asking because I bought the Ikea Parasoll door sensor and they’ve been flawless with the Sonoff Zigbee Dongle E + Ember Firmware + Zigbee2MQTT. Maybe you could buy one of those and see if it work any better for you.