Building a Jarvis-inspired voice activated LLM powered virtual assistant

I’d like my computer to be smarter and more interactive and handle boring stuff for me and I’d also like to play around with some LLM / AI stuff… which brings me to this project. I’ve got a ton of basic things I’d love for it to do – manage lists, reminders, some Outlook functions, some media functions, and then also be able to interact with me – all via voice commands. Yes, you can do this with ChatGPT and probably others – but I am loathe to provide any outside resource with more of “me” (DNA, biometrics, voice, ambient noises, etc) than absolutely necessary. Plus, I’ve been tinkering with these little LLM’s for a while now and see just what I can build out of them and with their assistance.

I’m not great at Python1 , so I admittedly enlisted the help of some very large LLM’s. I started the main project in conjunction with ChatGPT, used Gemini to answer some basic questions about programming in Python syntax, etc, and Claude for random things. The reason for keeping my general questions in Gemini versus ChatGPT was so that I could not “pollute” the ChatGPT flow of discussions with irrelevant sidetracks. This was the same reason for separating out the Claude discussions too. I find Claude reasonably helpful for coding tasks, but the use limits are too restrictive.

My kiddo asked me how much of the code was written by these models versus my own code. I’d say the raw code was mostly written by LLM’s – but I’m able to tinker, debug, and… above all learn. I’d rather be the one writing the code from scratch, but I’m treating these LLM’s like water wings. I know I’m not keeping myself fully afloat – but I’m actually the one treading water, putting it all together, and learning how to do it myself. Also… said kiddo was interested in building one too – so I’m helping teach someone else manually, and learning more that way.2

Ingredients

As with many of projects, I started by testing the individual pieces to see if I could get things working. In order I started with validating individual pieces of the process:

Could I get Python to record audio?
- Yep! Using import sounddevice, soundfile!
Could I get Python to transcribe that audio?
- Yep! Using speech to text and Whisper (well, Whisper small)
Could I get Python to use an API to run queries in LM Studio?
- Yep! Using the openai API, I could use python to send queries to LM Studio after an LLM had been loaded into memory
Could I get Python to get my computer to respond to a “wakeword”?
- Yep! There’s another Python module for using “wakewords” using PocketSphinx. This was an interesting romp. I found that I had to really tinker with the data being sent to the Wakeword to be properly recognized and then fiddle with the timing to make sure what came after the wakeword was properly captured before being sent to the LLM. Otherwise, I ended up with “Jarvis, set a timer for 15 minutes” would become… “Jarvis, for 15 minutes” since the “Jarvis” would get picked up by the wakeword but the rest not caught in time to be processed by whisper.
Can I get Python to verbally recite statements out loud?
- Yep! I used text to speech using Piper. However, this process took a while. One thing I learned was that you needed not just the voice model’s *.ONNX file, but the *.JSON file associated with it.

Until this point, I had wanted to try running LLM’s with the training wheels from LM Studio’s API. I really like the LM Studio program, but I don’t want to be dependent upon their service when I’m trying to roll my own LLM interface. Python can run LLM’s directly using “llama-cpp-python” – except that it will throw errors on the version of Python I was running (3.14) and was known to work with a prior version (3.11).

This lead me to learning about running “virtual environments” within Python so that I can keep both versions of Python on my computer, but basically run my code within a specific container tied to the version I need. Typing this command created the virtual environment within my project folder. The second command will “activate” that virtual environment.

py -3.11 -m venv venv
- This created the virtual environment, locked to Python 3.11
.venv\Scripts\activate
- This activates the virtual environment, so I can start working inside it

Back to work!

The man's got a job to do — The man’s got a job to do

Building a Pipeline

This is where things really seemed to take off. I was able to disconnect my script from LM Studio and use Python to directly call the LLM’s I’ve downloaded. These were reasonably straightforward – and I was suddenly able to go from: Wakeword -> whisper transcribed LLM query -> LLM response -> Piper recited reply. Then, it was reasonably easy to have the script listen for certain words, and perform certain actions (setting timers was the first such instance).

Optimizations, Problems, Solutions

Building something that kind worked brought me to a new and interesting ideas, challenges, and problems:

The original cobbled together process was something like: record audio, transcribe through Whisper, delete the recording, pass the transcribed statement to the LLM, give that statement to Piper, generate a new recording, play that recording. However, this process has some obvious “slop” where I’m making and deleting two temporary audio files. The solution was to find ways to feed the recording process directly into Whisper and feed Piper’s response directly to the speakers, cutting out the two audio files.
I realized that I wanted the script to do more than just shove everything I have to say / ask into an LLM – to be really useful, the script would have to do more than just be a verbal interface for a basic LLM. This is where I started bolting on a few other things – like trying to call a very small LLM to try and parse the initial request to either:
1. Something that can be easily accomplished by a Python script (such as setting a timer)
2. Something that needed to be handled by a larger LLM (summarize, translate, explain)
3. Something that maybe a small model could address easily (provide simple answer to a simple question)
I ran into some problems at this point. I spent a lot of time trying to constrain a small LLM3 to figure out what the user wanted and assign labels/tasks accordingly. After a lot of fiddling, it turns out that an LLM is generally a “generative” model and it wants to “make” something. My trying to force it to make a choice among only a dozen “words”4 was really bumping into problems where it would have trouble choosing between two options, choose inconsistently, and sometimes just make up new keywords. Now, I could come up with a simple Python script which just did basic word-matching to sort the incoming phrases – but it seemed entirely counterproductive to build a Python word-matching process to help a tiny AI. I then tried building a small “decision tree” of multiple small LLM calls to properly sort between “easy Python script call” and “better call a bigger LLM to help understand what this guy is talking about” and quickly stopped. Again, my building a gigantic decision tree out of little LLM calls was proving to be a bigger task, adding latency and error with each call. I was hoping to use a small LLM to make the voice interaction with the computer simple and seamless and then pass bigger tasks to a larger LLM for handling, sprinkling in little verbal acknowledgements and pauses to help everything feel more natural. Instead I was spending too much time building ways to make a small LLM stupider, doing this repeatedly, and then still ending up with too much slop.
And, frankly, it felt weird to try and lobotomize a small LLM into doing something as simple as “does the user’s request best fall into one of 12 categories?” Yes, small LLM’s can easily start to hallucinate, they can lose track of a conversation, make mistakes, etc. But, to constrain one so tightly that I’m telling it that it may only reply with one of 12 words feels… odd?

Tell me what I want to hear and this can all stop

Over the last few days I’ve been tinkering with building an “intent classifier” or “intent encoder” to do the kind of automatic sorting I was trying to force an LLM to do. As I understand this process, you feed the classifier a bunch of example statements that have been pre-sorted into different “intent slugs.” The benefit of a classifier is that it can only reply with one of these “intent slugs” and will never produce anything else. It’s also way faster. Calling a small5 LLM with a sorting question could produce a sometimes reliable6 answer in about 0.2 ms, which is almost unnoticeable. Calling a classifier to sort should enable a 97% reliable result within 0.05 ms. This is so fast it is imperceptible.

I haven’t tried this yet. I’ve built up a pile of “examples” from largely synthetic data to feed into a classifier, produce an ONNX file7 , and try out. However, I wanted to pause at this juncture to write up what I’ve been working on. I say synthetic data because I didn’t hand write more than 3,000 examples on some 50 different intent slugs. I wrote a list of slugs, described what each one should be associated with, created a small set of examples, and then asked Gemini to produce reasonable sounding examples based on this information. 8 This list appeared pretty good – but needed to be manually edited and also tidied up. I wanted to remove most of the punctuation and adjust the ways numbers and statements showed up, because I’m simply not confident that Whisper will be able to accurately match “Add bananas to shopping list” to “Add bananas to ‘shopping list'” to something that the classifier will correctly interpret.

As I tinker with this project… I’m also looking at how I might be able to extend it into further projects. Not only might it be a great way to help me be more productive, but I might be able to create a really small version that could be put into a companion bot. A little companion bot with limited space, power, inputs, and abilities to emote could be far more lifelike, independent, and non-deterministic in it’s responses and actions.

Project Jarvis

Building a Jarvis-inspired voice activated LLM powered virtual assistant

Yet!! [↩]
Thanks Mr. Fenyman! [↩]
Giving it limited context windows, limited tokens to use, highly restrictive system prompts [↩]
Make timer, list timers, make a reminder, add to a list, recite a list, media buttons, etc [↩]
~1B parameter [↩]
Let’s say 65% reliable [↩]
Yes! Just like the voice models!! [↩]
I know, more self-reflecting LLM garbage… [↩]

Python Practice with an LLM

I’ve been tinkering with Python more recently. When used on a MCU1 or a PC, it’s such a nice experience being able to write some code, run it without having to compile, see what happens, and adjust as necessary. Now, since I’m a newb at this, I’m getting help from… *shudder* LLM’s.2 Now, in the past I’d turn to Googling, looking at reliable and friendly forums such as Adafruit and Arduino, but I’d invariably need to check out Stack Overflow as well.3

As you might imagine, Stack Overflow was something of a victim of it’s own success. It’s content was good enough to train the LLM’s of the world – and those LLM’s can parrot / offer all the insights gleaned from Stack Overflow without the caustic haughty condescending replies typical of the comment sections on Stack Overflow / SlashDot / HackADay. Thus, it’s no small wonder the following graphic was circulating on Reddit:

Where was I? Oh, yeah… I was using some LLM’s to help with Python. I don’t have any fancy GPU’s, BitCoin mining rigs, etc, so I’m just using my non-gaming PC’s modest 16 GB VRAM to run the smaller local LLM’s. I can run things up to about 8B parameters, like the various Llama flavors, at 8 bit quantization with reasonable speed. I’ve found for my system that Qwen3 4B to be fast, thoughtful, and helpful.

I’ve realized this blog post is woefully low on actual Python related content. Here’s some things for future-me to remember:

pip list
- Will give me all the names of all packages installed
pip install requests Pillow reportlab PyPDF2
- Will install multiple packages, one after another

Python Programming Practice

Python Practice with an LLM

Microcontroller unit [↩]
Large language models such as [↩]
I bought their April Fool’s joke keyboard turned real product and once I’d remapped the keys, got significant use out of it for a long time. Between the construction, packaging, and accessories, at $30 this is still a total no-brainer if you need a small extra keyboard dedicated to some specific tasks. [↩]

Coding with an LLM Sidekick

I fell down a rabbit hole recently which lead me to think about my experiences in the nascent field of “prompt engineering.”1 2

As a thought experiment, I was thinking about what I’ve managed to accomplish working with an LLM, the challenges along the way, and perhaps even where I can see the frayed edges of its current limitations.

After several starts and stops trying to hire someone to assist with a website I own, I turned to the idea of getting help from an LLM. 3 4 After all, some of them were touted as being able to actually draft code, right? Besides, if the first step in even hiring a developer is just being able to describe what you need, and the first step of getting an LLM to generate some code is defining what I need, then…

There's no way this is going to work, right? — There’s no way this is going to work, right?

Task 1: Pie Chart WordPress Plugin
1. I started off with a simple and easy to define task. My original plugin was a quick and dirty bit of code, so if ChatGPT could create a WordPress plugin, there was a chance it could do something simple like this.
2. My first attempt was a wildly spectacular, but highly educational, failure. A brief description of the plugin’s function was enough to get a WordPress plugin template file with very little functionality. Then came the arduous LLM wrangling, my asking it for refinements, it losing track of the conversation, and the endless sincere heartfelt apologies from ChatGPT about forgetting really basic pieces of information along the way. Some changes were minor, but changing the names of variables, functions, the plugin, switching API’s, forgetting requirements, etc. It was constant whack-a-mole that spanned nearly 90 pages of text.
3. My next attempt was more focused. I created a framework for discussions, provided more context, goals, descriptions of workflow, and resources for examples. The result was a lot better, with portions of largely functional code. However, the LLM kept forgetting things, renaming variables, files, directories, etc.
4. Next I created the directory structure and blank placeholder files, zipped these, and uploaded them as an attachment for the LLM to review – along with a description of the contents and the above additional context. This was even better than before, but after a certain depth of conversation no amount of reminding could bring the LLM around to the core of the conversation.
5. My thinking was that after a certain level of conversation, the LLM was not going to be able to synthesize all of nuance of our conversations plus the content of the code drafted. To get around this I would begin a conversation, make a little progress, then ask it to summarize the project, the current status, and a plan for completion – which was fed into an entirely new conversation. This way, Conversation N was able to provide a succinct and complete description which Conversation N+1 could use as a jumping off point. My thinking was that the LLM would be best positioned to create a summary that would be useful to another LLM.
6. This process of minor “restarts” in the conversation was one of the most successful and powerful techniques I’ve employed to combat LLM hallucinations and forgetfulness.
Task 2: Blog Post Series Plugin
1. After rewriting the above pie chart plugin using an LLM, I turned my attention to a slightly more complicated plugin. The pie chart plugin is really just a single file which turns a shortcode with a little bit of data into a nice looking pie chart. There’s no options page, no cross post interaction, database queries or anything. It was really just a test to see if an LLM could really draft a basic piece of working code.
2. The series plugin is still a reasonably simple piece of code, but it has several additional feature which require a settings page, saving settings, custom database queries, and organizing information across multiple pages. It’s also one of the most used plugins on this website.
3. I figured I would try feeding the LLM a description of my plugin, all the code in a directory structure, and then my initial “base” prompt which explains our roles, needs, resources, and scaffolding for a discussion. I asked the LLM to summarize the function and features of the plugin, which it did quite nicely. I added a few additional features I had previously worked on and asked it to incorporate this into the description. Asking the LLM to simply “build this WordPress plugin” was met with a “you need to hire a developer” recommendation. However, asking it to propose a workflow for building a plugin with these features was successful. I was provided with a roadmap for building5 my plugin.
4. This system worked reasonably well, allowing me to compartmentalize the steps, backtrack, retrace, revise code, working on a section, then another, sometimes going back to a prior sections at the LLM’s direction. The LLM still tended to get lost, renamed variables/paths/directories/filenames, but it was less pronounced than before. I did find it harder to use the “summarize and restart” strategy when dealing with a multi-step code development system. However, it was still workable since I could upload all the code produced so far.
5. The result was a new plugin, with better functionality than what I’d written myself 10 years before. Here, the new strategy of having the LLM break the project into sections and providing a roadmap was particularly helpful.
Strategy: Conversational Scaffolding
1. I mentioned “conversational scaffolding” and “frameworks” for discussing things with the LLM above. This was an overarching and evolving strategy I use to help focus the LLM on the goals, keep it on track, and hopefully help it provide meaningful and useful replies. The full text of my “prompt framework” file is too large to include here, but I’m happy to provide the highlights.
2. Personas. I assigned the LLM three distinct personas with differing backgrounds, strengths, and goals. Their personas were defined in reference to one another, so the first would activate, the second would then review and interact with the first, after this process completed the third would be activated, perhaps interact with the first two, then it would move on. I would say this process was rather successful.
3. Myself. I would describe myself, my goals, level of expertise, etc. I found that I if I referred to myself as an expert, the LLM would not be as likely to offer me code proposals – but if I described myself as a newbie, it would recommend I hire a developer rather than tackle such a complex problem myself.
4. Rules for Conversation. These are a collection of 12 rules (at last count) which helped myself and the LLM interact. The high points are:
  1. Answer Numbering, Answer Format, Eliminate Guesswork, Organize Assumptions, Conversational Review, Complex Answers, Context Refresher, Problem Solving Approach, File Structure, @Rules, and Personas.
  2. Each of these items were followed by a few sentences explaining something about how the LLM should be expecting to receive information and react. My favorite of these was the rule “@Rules” which directed the LLM to begin it’s response by reviewing the Rules and following them.
5. Knowledge. There are a number of programming languages and technical topics I’m interested in and have used an LLM to address. To this point, I solicited a list of useful resources from the LLM and started including a “Knowledge” section where I listed dozens of the most important resources for the languages and API’s I most commonly use.
6. By beginning each prompt with the above “framework” (~10k of text) and following it up with a short description of my project or a file to consider, I found I was able to jump right into the project without having to provide additional significant background information.
Task 3: “Project Drift”
1. This is a considerably more complicated task I will simply refer to as “Project Drift.” This isn’t a real codename since the developer base is all of exactly one dude, but I don’t want to name the location/website for a variety of reasons. In any case, Project Drift involves multiple user interfaces, numerous settings, database queries, data sanitation and validation procedures, administrator functions, and numerous other facets. All of the above tasks and attempts were basically part of the run-up to this (ongoing) project.
2. Using the LLM’s ability to open and read a ZIP file, as well as propose code, has been invaluable. This in conjunction with my prompt framework allows me to get the LLM up to speed after a micro-restart – and it’s summarization procedures help me get back in the mindset after I’ve stepped away from the project for a few days.
3. Since this project isn’t done yet, I can only give a progress report. It’s going very well. Much of the heavy lifting, scaffolding of the code, can be assembled for me, tedious database queries and chunks of code provided. There are still large areas where the LLM is unable to be very helpful – and that relates to pinpointing a bug in the code (or between code sections). This still requires a knowledgeable hand at the helm.
4. As a solo-coder, having the assistance of another “persona” to keep me on track with a given section of code has been helpful. I have only assigned three personas, but I could see adding a few more to fulfill different roles.

I would estimate Project Drift is roughly 30-50% complete, but this is still an incredible amount of progress in a very short time. I would also estimate it has cut the amount of my development time by 90% (but on the easiest and most tedious stuff).

Software Development with LLMs

I know, it feels pretentious, doesn’t it? [↩]
I’ve got the same knee-jerk reaction to “visionary,” “thought leader,” “polymath,” and “futurist.” [↩]
Don’t get me wrong, some of the developers I’d hired simply disappeared while other relationships didn’t work out due to timing. I don’t think anyone was malicious, just… busy, really. [↩]
Still, the job needs to be done. [↩]
Re-building? [↩]

ChatGPT WordPress Plugins

This is kinda bananas. Years ago I wrote a plugin to solve a problem I had. I wanted a simple WordPress plugin where I could insert a shortcode into a blog post, specify a series title, and have it automatically search up all the other blog posts that used the same shortcode and series title, and then insert a nice looking list of blog posts in that series in chronological order.

It was one of my first plugins, still available on WordPress.org – just hidden since it hasn’t been updated in almost a decade. It still works to this very day, if occasionally a little buggy. After several WordPress versions, it no longer properly displays the series title, which is a real shame.

On a whim, I tried using ChatGPT to generate some plugins.

Here’s an example of my old plugin and the new ChatGPT written plugin (in this order):

Default Series Title

See how bad that was? It completely mangled the title.

Edit: Since publishing this post, I realized that I would have to choose between

Leaving the old defunct plugin in place just to make a point about how it didn’t stand the test of a decade’s worth of WordPress updates, but then also leaving broken series titles sprinkled through my back catalog of blog posts.
Go back through nearly 10 years of blog posts1 2 to change them over to the new plugin shortcode.
Disable the old plugin, but have the new plugin work with the old shortcode as well as it’s own new shortcode, at the cost of losing an example of how bad the old plugin performed.

I went with option 3. Just take my word for it, it looked bad.

Now for the ChatGPT version:

Software Development with ChatGPT

ChatGPT WordPress Plugins

It took me about an hour to whip up a working WordPress plugin with the same core functionality. I would break down the time I spent as follows:

Time Spent Creating Series Plugin with ChatGPT

But, that’s not all! You see, as I was writing this blog post, I realized it would be fun to include a pie chart to indicate the time I’d spent on this. Unfortunately, the plugin I had written to do exactly this many years ago has apparently completely given up the ghost. Thus, before I proceeded to this very sentence, I used ChatGPT to create a plugin for displaying custom pie charts!

Time Spent Creating Pie Chart Plugin with ChatGPT

Obviously, this plugin took a lot longer. The first few versions were having all kinds of problems between the HTML Canvas code and trying to figure out how to make sure the javascript was not loading too early or too late. In the end, I just asked it whether it was capable of even creating a pie chart – and it gave me a piece of workable javascript. I told it to refactor the plugin using this same javascript, and then it was a matter of fine tuning the result.

If you don’t know anything about writing WordPress plugins, you could probably use ChatGPT to create a very simple plugin. However, once it got slightly more complicated, it would likely require some troubleshooting to figure out what was happening. In the series plugin it took me a while to root through the WordPress functions to figure out that apparently ChatGPT was trying to use a function in a way that simply did not work. I explained to ChatGPT that that particular function could not operate in that way, explained how the data it was feeding into that function needed to be first modified, and then asked it to refactor the code. From that point forward, it started to look a lot better. There were some additional quirks – like putting more than one series title in a single post would only display one. I suspect these problems of ChatGPT taking a shortcut to generate code, hardcoding certain variables and names, not considering that it might need to operate more than once on the page, may be difficult for it to anticipate and address. Without some degree of WordPress development knowledge, I think a novice user armed only with ChatGPT would need to do a lot of refactoring, asking the program to generate the plugin all over from scratch many times, before arriving at a workable result. Then again, a million monkeys at typewriters, right?

I think ChatGPT could be great for creating relatively simple plugins like a series plugin, a pie chart plugin, or even a table of contents plugin. However, having seen how much time it cut out of the development process, I think it would be interesting to try developing an A/B testing plugin or more complicated plugin.

I think the next task to see if I can get it to generate QMK code for a keyboard, Arduino code, Raspberry Pi code, or a chrome extension.

I can already see some ways to improve both of the ChatGPT generated plugins used in this blog post. My series plugin included two arrows at the bottom so the reader could navigate to the prior or next post in the series. And I think it would be great if the chart plugin had a feature where I could specify the units, so the magnitude data would be included with the labels. I may try getting it to shoehorn these updates later…

If you see these reflected in the charts above, I must have already done it. :)

Software Development with LLMs

NGL, I can really be a lot some times. [↩]
Um, you’ve probably gathered that. [↩]

Series Plugin Test for Illustrative Purposes Only

The only purpose for this post is to serve as a reference for a more interesting and useful post.

Software Development with LLMs

[custom_pdf_generator visitor_data=“John Doe”]

Ingredients

Building a Pipeline

Optimizations, Problems, Solutions

Task 1: Pie Chart WordPress Plugin

Task 2: Blog Post Series Plugin