The article discusses the Jev framework and related self-hostable projects, including OpenJev and SemIf. It highlights a technique for reading an LLM's token probabilities, referencing OpenAI's logprobs cookbook. The author explains how to structure a prompt and add JSON request parameters to a Chat Completions request, which allows the LLM API to return the letter along with the model's log probabilities for alternative tokens. This method is noted for its efficiency, as generating only one token results in quicker responses.
The author also mentions that this technique can be applied to vision models, although Jev's current documentation primarily covers text/JSON state. The author added an attachments field for images during local experiments, capturing webcam frames and sending base64 JPEGs. The experiments included questions about visibility, location, and brightness of the scene. The author reports achieving approximately one frame per second with the Gemma 4 12B model on an RTX 3090, and around 0.2 frames per second with OpenAI's gpt-6-luna model due to connection costs.
The article concludes with a Python example script that manages API differences between llama.cpp and OpenAI, demonstrating how to access the webcam and process the data.