Evaluating Large Language Models Trained on Code

I created a basic proof of concept of LMVM (Language Model Virtual Machine), a command line toolchain that English instructions into native binaries without intermediary high-level language compilation. The command line uses a large language model (LLM) as IR generation backend, producing LLVM Intermediate Representation (IR) directly from user instructions coding. The IR is compiled and linked to machine code (-o files) via `clang` (with an `llc`/`lld` fallback on POSIX hosts) and executed on bare metal, within Docker sandboxes, on AWS Lambda, or as standalone HTTP servers. How It Works (5 Steps) 1. You type an instruction - e.g. "Create a rest api running on localhost port 3000 and return hello world" 2. LMVM checks your computer - It figures out if you're on Windows/Mac/Linux, what CPU you have (Intel vs ARM), and makes sure clang (the compiler) is installed. If not, it offers to install it for you (with your permission). 3. An AI writes LLVM IR - Instead of writing C or Python, the LLM writes LLVM IR -- a low-level assembly-like language that sits between source code and machine code. The prompt includes your OS details so the AI generates code that works on your platform (e.g. Winsock on Windows vs POSIX sockets on Mac/Linux). 4. Clang turns IR into a native binary - clang compiles the IR and links it with system libraries (PostgreSQL, OpenSSL, SQLite, etc.) into a real executable. One step, no extra tools needed. 5. It runs - The binary executes directly on your CPU. If it's a Docker target, it runs sandboxed. If it's Lambda, it packages for AWS. 2. Why we care : Compiled native binaries outperform interpreted languages by 10-50x in CPU usage and 5-15x in memory for compute-heavy tasks, because interpreters add overhead with every instruction they process and their single-threaded execution model blocks true parallelism. Few runtimes that compile code on the fly, the gap narrows to 2-5x in CPU since those systems optimize hot code paths over time -- but native code still starts faster, uses less memory, and can run true parallel threads instead of juggling everything through a single event loop. The biggest gains show up in recursive and number-crunching workloads (where the cost of "figuring out what to do next" dominates) and memory-heavy tasks (where automatic garbage collection and wrapping every value in an object bloats memory usage). I/O-bound tasks show the smallest differences, because the operating system does the real work regardless of what language runtime sits on top. This also means we need to think carefully about maintainability, especially now that AI agents can generate code as a complete black box sometimes even producing low-level output that humans rarely inspect. 3. System Architecture [CLI] → [Toolchain] → [LLM Client] → [Compiler] → [Runner] → Output ↑ | Error Feedback | Self‑Correction Loop (x10) 3.1 CLI and Interactive Setup The command-line interface (built on Commander.js) exposes four commands: `run` (compile and execute), `compile` (IR generation only), `build` (compile existing IR), and `targets` (list output targets). 3.2 LLM Client The LLM client wraps an OpenAI-compatible API, supporting providers examples (OpenAI, Groq, Together AI, OpenRouter, and local llm instances). The command line agent constructs prompts from two sources: a system prompt (loaded from `Prompt.md`, ~4000 tokens) containing LLVM IR generation rules, security constraints, and code templates; and a user prompt containing a platform descriptor (host OS, exact target triple, sockets dialect), the instruction, target, and error context for retry attempts. A secondary method, is used by the toolchain bootstrapper to obtain package-manager commands for the detected host. 3.3 Platform Abstraction and Toolchain Bootstrapper Since LMVM builds real native programs, it has to know exactly what computer it's running on -- the same code won't work the same way on Windows vs Mac vs Linux. So a platform module figures out three things: 1. The "target triple" -- a label that tells the compiler exactly which OS and CPU to build for (like x86_64-pc-windows-msvc for 64-bit Windows, or arm64-apple-darwin for an M-series Mac). 2. How to do networking -- Windows uses its own socket system (Winsock), while Mac and Linux use the standard POSIX approach. The code has to match. 3. What package manager is available -- winget/choco/scoop on Windows, brew on Mac, apt/dnf/pacman on Linux. This matters if LMVM needs to install something. All of this gets described in plain text and injected into the AI's prompt, so it writes code that actually compiles on your machine instead of assuming you're on Linux. Before talking to the AI, LMVM runs a checklist to make sure your computer can actually compile things: 1. PATH fix - If you just installed clang but your terminal hasn't refreshed yet, LMVM detects the install location and patches its own PATH so it finds clang right away, no need to restart your terminal. 2. Is clang there? - It checks that the compiler is actually available. 3. Can clang actually build a program? (Windows only) -- On Windows, having clang isn't enough -- it also needs the MSVC linker and Windows SDK behind it. LMVM compiles a tiny test program to verify the full chain works. If it fails, the Visual Studio Build Tools are probably missing. 4. Install with user's permission - If anything is missing, LMVM asks the AI for the right install commands, shows them to you, and waits for your yes before running anything. Then it checks again. If the toolchain still can't work after all that, LMVM stops early and tells you exactly what to do (install manually, open a new terminal, or use Docker instead). It doesn't waste your AI API calls retrying something a code fix can't solve. 3.4 Code Compiler Compilation is driven by `clang`, which lowers the `.ll` and invokes the platform linker in a single step: Primary path (all platforms) - `clang output.ll -o `. Library flags are platform-aware: POSIX hosts use `-lpq`, `-lcurl`, `-lssl`, etc.; Windows links `ws2_32` for socket/server targets and relies on the MSVC C runtime. For the AWS Lambda target, static linking is applied. POSIX fallback - If `clang` is unavailable on macOS/Linux, the system falls back to the classic cascade: the LLVM Static Compiler (`llc -filetype=obj -relocation-model=pic`) to produce a position-independent object file, then `ld.lld` (`-flavor gnu`), the system C compiler (`cc`), or `clang` as a linker. A 60-second timeout bounds each invocation. The compiler maps library names to their respective linker flags through platform-specific lookup tables, covering PostgreSQL (libpq), HTTP (libcurl), JSON (cJSON), SSL/TLS (OpenSSL), image processing (libpng, libjpeg), databases (libsqlite3, libmysqlclient), compression (zlib), encryption (libcrypto), caching (hiredis), and message queues (librdkafka). 3.5 Execution Runner The runner dispatches binary execution based on the detected target like native , docker or AWS or GCR. 3.6 Self-Correction Loop When compilation or execution fails, the runtime updates a VM state object (`VMState`) with the error type (compiler/linker/runtime), error message, error line number, previous IR, and a compressed memory object (`VMMemory`) tracking accumulated errors and resolved dependencies. This enriched context is fed back to the LLM in the next attempt, enabling targeted correction. The loop terminates after a configurable maximum number of attempts (default: 10). 3.7 Security Layer LMVM employs defense-in-depth across three layers: 1. AI rules -- The instructions sent to the AI explicitly say: don't generate code that runs shell commands with user input, don't write files outside the allowed folder, don't read sensitive system files, don't hardcode passwords, and don't allocate unlimited memory. Think of this as a "don't do bad things" rulebook for the AI. 2. Sandbox -- When running in Docker, the program is locked down. A seccomp profile blocks the most dangerous system calls -- things like creating new processes, mounting drives, rebooting the machine, loading kernel modules, or snooping on other programs. Docker also limits the program to 512 MB of RAM and 1 CPU core, so it can't eat all your resources. This follows NIST security guidelines. 3. Input protection -- When LMVM runs the compiler or your program, it passes arguments as a structured list, not a single string. This means a hacker can't sneak in shell commands the way they might with string concatenation (the classic "shell injection" attack). The one exception is the toolchain installer, which does run shell commands -- but it always shows you the exact commands first and waits for your approval before running them. API keys are hidden in console output, and data buffers are capped at 10 MB to prevent memory explosions Appendix : As a demo using command LMVMcli command line we created a basic rest api directly as a machice code. Generated IR by GPT