I built this in parts so I could understand the system instead of treating it as a black box. Each container has a specific role, and the storage\/backups are designed so the system can be rebuilt later if something fails.<\/p>\n\n\n\n
The larger goal is to turn this machine into a private multimodal AI workstation over time. For now, this post focuses on the infrastructure foundation: the host, the containers, the models, the performance notes, the troubleshooting, and the backup strategy.<\/p>\n\n\n\n
\n\n\n\n
Project Goals<\/h2>\n\n\n\n
My main goals for this project were:<\/p>\n\n\n\n
- \n
- Build a local AI system on my own hardware<\/li>\n\n\n\n
- Separate the main AI services into containers<\/li>\n\n\n\n
- Use consumer NVIDIA GPUs for local inference experiments<\/li>\n\n\n\n
- Test practical backends like Ollama and llama.cpp<\/li>\n\n\n\n
- Experiment with vLLM, FP8, and multi-GPU serving<\/li>\n\n\n\n
- Keep large model files outside the container root disks<\/li>\n\n\n\n
- Create hot\/warm backups on local storage<\/li>\n\n\n\n
- Plan for cold storage backups on an external drive<\/li>\n\n\n\n
- Document the setup well enough that I could rebuild it later<\/li>\n\n\n\n
- Turn the build into a portfolio project instead of leaving it as undocumented experimentation<\/li>\n<\/ul>\n\n\n\n
I also wanted the system to support future projects, especially a Hermes-style local agent workflow, cybersecurity\/IT support assistants, and eventually local multimodal workflows such as image, speech, transcription, and video processing.<\/p>\n\n\n\n
\n\n\n\nVisual Stack Model<\/h2>\n\n\n\n
This is the current stack at a high level:<\/p>\n\n\n\n
User \/ Browser \/ Main PC\n |\n v\n+------------------------------+\n| CT100: OpenWebUI + Ollama |\n| - Main web interface |\n| - Daily-use AI frontend |\n| - Ollama model backend |\n+--------------+---------------+\n |\n | OpenAI-compatible \/ backend connections\n v\n+------------------------------+ +------------------------------+\n| CT101: llama.cpp | | CT102: vLLM |\n| - Qwen3.8 GGUF backend | | - Qwen3.8 FP8 proof of |\n| - Stable API backend | | concept |\n| - Agent-backend candidate | | - Short-context testing |\n+------------------------------+ +------------------------------+\n\nHost Platform:\n+--------------------------------------------------------------+\n| prox0 - Proxmox VE |\n| - Ryzen 9 5900X |\n| - RTX 5060 Ti 16GB + RTX 3060 12GB + RTX 3060 12GB |\n| - NVMe ZFS mirror for Proxmox and CT root disks |\n| - SATA AI storage for models, cache, docs, and warm backups |\n+--------------------------------------------------------------+\n\nStorage and Recovery:\n+---------------------+ +----------------------------+\n| NVMe ZFS mirror | | \/mnt\/ai-sata SATA storage |\n| - Proxmox OS | | - Ollama models |\n| - CT root disks | | - GGUF models |\n| | | - vLLM HF cache |\n| | | - warm CT backups |\n| | | - documentation |\n+---------------------+ +----------------------------+\n |\n v\n External HDD \/ Cold Storage\n - disaster recovery copy<\/code><\/pre>\n\n\n\n![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-205804-1024x531.png\")
<\/figure>\n\n\n\n
\n\n\n\n![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-205932.png\")
<\/figure>\n\n\n\n
\n\n\n\nPart 1 \u2014 Host Hardware and Proxmox Foundation<\/h2>\n\n\n\n
The host is named prox0<\/strong> and runs Proxmox VE. It is built on a Ryzen 9 5900X with about 32 GB of RAM and three NVIDIA GPUs:<\/p>\n\n\n\n
- \n
- RTX 5060 Ti 16 GB<\/li>\n\n\n\n
- RTX 3060 12 GB<\/li>\n\n\n\n
- RTX 3060 12 GB<\/li>\n<\/ul>\n\n\n\n
That gives the rig about 40 GB of installed VRAM across the system. One important point I learned is that total VRAM is not the same thing as one large shared memory pool. Each backend handles multi-GPU workloads differently, and some models split better than others.<\/p>\n\n\n\n
The current philosophy is to run one heavy AI workload at a time<\/strong>. This is not a production cluster meant to keep every model loaded at once. It is a local AI workstation where I can choose the best backend for the job.<\/p>\n\n\n\n
![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-210125-1024x545.png\")
<\/figure>\n\n\n\n
\n\n\n\nPart 2 \u2014 Storage Design<\/h2>\n\n\n\n
I separated the storage into two main layers:<\/p>\n\n\n\n
NVMe ZFS Mirror<\/h3>\n\n\n\n
The NVMe mirror is used for:<\/p>\n\n\n\n
- \n
- Proxmox OS<\/li>\n\n\n\n
- Container root disks<\/li>\n\n\n\n
- Core system state<\/li>\n<\/ul>\n\n\n\n
This gives the host and container root filesystems better protection than a single boot drive.<\/p>\n\n\n\n
SATA AI Storage<\/h3>\n\n\n\n
Large AI files live on a SATA SSD mounted at:<\/p>\n\n\n\n
\/mnt\/ai-sata<\/code><\/pre>\n\n\n\nThis storage holds:<\/p>\n\n\n\n
- \n
- Ollama model store<\/li>\n\n\n\n
- GGUF models<\/li>\n\n\n\n
- vLLM Hugging Face cache<\/li>\n\n\n\n
- vLLM cache<\/li>\n\n\n\n
- Proxmox warm backups<\/li>\n\n\n\n
- Documentation files<\/li>\n<\/ul>\n\n\n\n
Important folders:<\/p>\n\n\n\n
\/mnt\/ai-sata\/ollama\n\/mnt\/ai-sata\/gguf\n\/mnt\/ai-sata\/vllm\n\/mnt\/ai-sata\/containers\/dump\n\/mnt\/ai-sata\/docs<\/code><\/pre>\n\n\n\nThis design keeps large model files out of the CT root disks. It also makes the backup plan more obvious: the container backup protects the container configuration and service environment, but the model folders need to be copied separately.<\/p>\n\n\n\n
![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-210533.png\")
<\/figure>\n\n\n\n
\n\n\n\nPart 3 \u2014 CT100: OpenWebUI and Ollama<\/h2>\n\n\n\n
CT100 is the main user-facing AI container. It runs OpenWebUI and Ollama.<\/p>\n\n\n\n
OpenWebUI gives me the browser-based interface, while Ollama handles local model serving. This is the easiest part of the stack to use every day.<\/p>\n\n\n\n
Main role:<\/p>\n\n\n\n
- \n
- Main local AI interface<\/li>\n\n\n\n
- Ollama backend<\/li>\n\n\n\n
- Daily-use candidate<\/li>\n\n\n\n
- Model storage through a SATA bind mount<\/li>\n<\/ul>\n\n\n\n
The Ollama model store is mounted from the host into the container:<\/p>\n\n\n\n
Host path: \/mnt\/ai-sata\/ollama\nContainer path: \/mnt\/ollama-models<\/code><\/pre>\n\n\n\nInstalled Ollama Models<\/h3>\n\n\n\n
At the time of documentation, the installed Ollama models were:<\/p>\n\n\n\n
Model<\/th> Size<\/th> Intended Role<\/th><\/tr><\/thead> qwen3.8:27b<\/code><\/td>17 GB<\/td> Main general-purpose local model for chat, reasoning, coding, research, and documentation help<\/td><\/tr> muse-glimmer:30b<\/code><\/td>18 GB<\/td> Creative\/image-related model candidate and future visual workflow testing<\/td><\/tr> nemotron-3.5-lightning:30b-a3b-q4_K_M<\/code><\/td>25 GB<\/td> Experimental alternate model, not necessarily a long-term keeper<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n The main daily-driver model direction is Qwen3.8 27B<\/strong>. My goal is not to collect many similar models. I want each model to earn a role.<\/p>\n\n\n\n
Observed Performance<\/h3>\n\n\n\n
In testing, Ollama with Qwen3.8 27B was the strongest daily-use result. A benchmark run showed roughly:<\/p>\n\n\n\n
Ollama Qwen3.8 27B\nPrompt eval: ~1,862 tokens\/sec\nGeneration: ~18.5 tokens\/sec<\/code><\/pre>\n\n\n\nThat made CT100 the easiest and most practical day-to-day interface.<\/p>\n\n\n\n
![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-220448-1024x485.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/image.png\")
<\/figure>\n\n\n\n
\n\n\n\nPart 4 \u2014 CT101: llama.cpp Backend<\/h2>\n\n\n\n
![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/ezgif.com-optimize-1.gif\")
<\/figure>\n\n\n\nCT101 runs llama.cpp and serves a Qwen3.8 GGUF model through
llama-server<\/code>.<\/p>\n\n\n\nThis container matters because llama.cpp gives me a stable OpenAI-compatible API endpoint. That makes it useful for OpenWebUI, future tools, and possible agent frameworks.<\/p>\n\n\n\n
Main role:<\/p>\n\n\n\n
- \n
- Stable OpenAI-compatible backend<\/li>\n\n\n\n
- GGUF model serving<\/li>\n\n\n\n
- Alternate daily-driver backend<\/li>\n\n\n\n
- Strong candidate for future Hermes\/agent integration<\/li>\n<\/ul>\n\n\n\n
Main model:<\/p>\n\n\n\n
\/models\/qwen3.8-27b\/Qwen3.8-27B-Q4_K_M.gguf<\/code><\/pre>\n\n\n\nThe model is stored on the SATA-backed
\/models<\/code> bind mount:<\/p>\n\n\n\nHost path: \/mnt\/ai-sata\/gguf\nContainer path: \/models<\/code><\/pre>\n\n\n\nllama.cpp Performance Notes<\/h3>\n\n\n\n
The llama.cpp backend performed well enough to be considered a real daily-driver candidate. In testing, the Qwen3.8 GGUF backend produced results around:<\/p>\n\n\n\n
llama.cpp Qwen3.8 GGUF\nPrompt eval: ~717 tokens\/sec\nGeneration: ~15.6 tokens\/sec<\/code><\/pre>\n\n\n\nOther runs showed generation in the mid-to-high 16 tokens\/sec range depending on the prompt and run conditions. The bigger takeaway is that CT101 was stable and practical, especially compared with the vLLM proof-of-concept.<\/p>\n\n\n\n
Notable llama.cpp Troubleshooting<\/h3>\n\n\n\n
One issue I documented was a
503 Loading model<\/code> API response immediately after service startup. That was not a failure. It meant the llama.cpp server had started, but the model was still loading. After startup finished, the API could be tested normally through\/v1\/models<\/code> and\/v1\/chat\/completions<\/code>.<\/p>\n\n\n\nThis was a useful reminder that service status and model readiness are not always the same thing. A systemd service can be running while the model is still warming up.<\/p>\n\n\n\n
![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-234512-1024x537.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-225558-1024x539.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-28-224017.png\")
<\/figure>\n\n\n\n
\n\n\n\nPart 5 \u2014 CT102: vLLM Proof of Concept<\/h2>\n\n\n\n
![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/ezgif.com-optimize-2.gif\")
<\/figure>\n\n\n\nCT102 is the vLLM experiment container. It is not my main daily-driver backend.<\/p>\n\n\n\n
The goal of CT102 was to prove that the rig could serve Qwen3.8-27B-FP8<\/strong> through vLLM using all three NVIDIA GPUs with pipeline parallelism.<\/p>\n\n\n\n
Main role:<\/p>\n\n\n\n
- \n
- vLLM proof-of-concept<\/li>\n\n\n\n
- Qwen3.8-27B-FP8 testing<\/li>\n\n\n\n
- OpenAI-compatible API endpoint<\/li>\n\n\n\n
- Multi-GPU pipeline-parallel experiment<\/li>\n\n\n\n
- Short-prompt demo backend<\/li>\n<\/ul>\n\n\n\n
Final served model:<\/p>\n\n\n\n
qwen38-27b-fp8<\/code><\/pre>\n\n\n\nHugging Face model:<\/p>\n\n\n\n
Qwen\/Qwen3.8-27B-FP8<\/code><\/pre>\n\n\n\nFinal Stable vLLM Profile<\/h3>\n\n\n\n
The final working vLLM profile used:<\/p>\n\n\n\n
pipeline_parallel_size = 3\ntensor_parallel_size = 1\nlayer split = 20 \/ 20 \/ 24\ncontext = 384 tokens\nKV cache = fixed 512 MiB\nKV cache dtype = fp8\nmode = text-only\neager execution = enabled<\/code><\/pre>\n\n\n\nThe important result is that CT102 works as a proof-of-concept. It proved that the rig could load and serve the FP8 model across all three GPUs.<\/p>\n\n\n\n
vLLM Performance Notes<\/h3>\n\n\n\n
One successful vLLM run showed approximately:<\/p>\n\n\n\n
vLLM Qwen3.8-27B-FP8\nPrompt throughput: ~39 tokens\/sec\nGeneration throughput: ~7.8 tokens\/sec<\/code><\/pre>\n\n\n\nThat result was useful, but it did not make vLLM the best daily workflow. The stable context was intentionally tiny, and the hardware layout created tradeoffs.<\/p>\n\n\n\n
Notable vLLM Troubleshooting<\/h3>\n\n\n\n
This was the most troubleshooting-heavy part of the project.<\/p>\n\n\n\n
Some approaches that did not work:<\/p>\n\n\n\n
- \n
- Tensor Parallel 3:<\/strong> failed because model dimensions\/vocab were not divisible by 3.<\/li>\n\n\n\n
- Tensor Parallel 2:<\/strong> could not fit cleanly on the two 12 GB GPUs without CPU offload.<\/li>\n\n\n\n
- CPU offload:<\/strong> technically helped memory pressure, but used too much RAM and hurt performance.<\/li>\n\n\n\n
- Larger context sizes:<\/strong> caused CUDA OOM, Marlin FP8 runtime failures, or EngineDead errors.<\/li>\n\n\n\n
- 21\/21\/22 layer split:<\/strong> shifted the failure point instead of solving it.<\/li>\n\n\n\n
- Mixed consumer GPU daily-driver use:<\/strong> FP8 vLLM was not a good daily-driver fit for the 12 GB + 12 GB + 16 GB GPU layout.<\/li>\n<\/ul>\n\n\n\n
The final stable profile was built by accepting the limitation and changing the goal: CT102 became a clean proof-of-concept instead of a daily-driver backend.<\/p>\n\n\n\n
![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-29-015058-1024x538.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-29-014858-1024x336.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/yairmartinezcybersecurityportfolio.com\/wp-content\/uploads\/2026\/09\/Screenshot-2026-09-29-015838-1024x731.png\")
<\/figure>\n\n\n\n
\n\n\n\nConsumer Hardware Tradeoffs<\/h2>\n\n\n\n
This project also showed the difference between consumer hardware that can run a workload and server hardware that is ideal for it.<\/p>\n\n\n\n
The system has three GPUs, but the PCIe layout is not perfect for multi-GPU serving. The two RTX 3060 cards were observed running at downgraded PCIe x1<\/strong> links, while the RTX 5060 Ti was observed at a downgraded x8<\/strong> link. That does not stop the system from working, but it matters for workloads that need GPUs to communicate or pass data through pipeline stages.<\/p>\n\n\n\n
The practical impact was clearest with vLLM. Pipeline parallelism across all three GPUs worked, but the x1 links on the RTX 3060s made this a better proof-of-concept than a daily-driver setup.<\/p>\n\n\n\n
This changed how I think about the backend roles:<\/p>\n\n\n\n
- \n
- Ollama:<\/strong> best convenience and daily-use experience so far<\/li>\n\n\n\n
- llama.cpp:<\/strong> stable OpenAI-compatible backend and strong agent candidate<\/li>\n\n\n\n
- vLLM:<\/strong> valuable experiment and portfolio proof, but not the main workflow on this hardware<\/li>\n<\/ul>\n\n\n\n
That lesson is important because it is easy to look only at total VRAM and miss the rest of the system design.<\/p>\n\n\n\n
\n\n\n\nBenchmark Snapshot<\/h2>\n\n\n\n
These numbers are not formal lab benchmarks. They are practical observations from my own testing and logs.<\/p>\n\n\n\n
Backend<\/th> Model<\/th> Approx. Prompt Throughput<\/th> Approx. Generation Throughput<\/th> Practical Result<\/th><\/tr><\/thead> Ollama<\/td> Qwen3.8 27B<\/td> ~1,862 tokens\/sec<\/td> ~18.5 tokens\/sec<\/td> Best daily-use result so far<\/td><\/tr> llama.cpp<\/td> Qwen3.8 27B GGUF<\/td> ~717 tokens\/sec<\/td> ~15.6 tokens\/sec<\/td> Stable OpenAI-compatible backend<\/td><\/tr> vLLM<\/td> Qwen3.8-27B-FP8<\/td> ~39 tokens\/sec<\/td> ~7.8 tokens\/sec<\/td> Working proof-of-concept, not daily driver<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n The benchmark results helped me make a practical decision. vLLM was interesting technically, but Ollama and llama.cpp were better fits for normal use on this rig.<\/p>\n\n\n\n
\n\n\n\nBackup and Recovery Plan<\/h2>\n\n\n\n
Backup planning became one of the most important parts of this project.<\/p>\n\n\n\n
The Proxmox CT backups protect the container root filesystems. That includes the installed services, configuration, scripts, and system state inside the containers.<\/p>\n\n\n\n
However, the large model folders are bind mounts. That means normal CT backups do not fully include the model directories.<\/p>\n\n\n\n
My simple way of thinking about the backup design is:<\/p>\n\n\n\n
CT backup = container brain \/ config \/ root filesystem\nModel folder backup = large model bodies and cache\nDocumentation = rebuild knowledge<\/code><\/pre>\n\n\n\nWarm Backup Layer<\/h3>\n\n\n\n
The SATA SSD acts as the warm backup\/staging layer. It stores:<\/p>\n\n\n\n
\/mnt\/ai-sata\/containers\/dump\n\/mnt\/ai-sata\/docs<\/code><\/pre>\n\n\n\nThe container backups are stored under:<\/p>\n\n\n\n
\/mnt\/ai-sata\/containers\/dump<\/code><\/pre>\n\n\n\nThis gives me a local recovery point if I break a container or need to roll back the AI services.<\/p>\n\n\n\n
Cold Storage Layer<\/h3>\n\n\n\n
The cold storage plan is to copy the important backup set to an external hard drive.<\/p>\n\n\n\n
Cold storage should include:<\/p>\n\n\n\n
\/mnt\/ai-sata\/containers\/dump\n\/mnt\/ai-sata\/ollama\n\/mnt\/ai-sata\/gguf\n\/mnt\/ai-sata\/vllm\n\/mnt\/ai-sata\/docs<\/code><\/pre>\n\n\n\nThis matters because a CT backup alone is not enough. If the SATA SSD failed and I only had the CT root backups, I would lose the downloaded model stores and caches.<\/p>\n\n\n\n
\n\n\n\nDocumentation Pack<\/h2>\n\n\n\n
I created a final documentation pack so the project can be understood later instead of relying on memory.<\/p>\n\n\n\n
The documentation pack includes:<\/p>\n\n\n\n
- \n
README.txt<\/code><\/li>\n\n\n\nai-rig-overview.txt<\/code><\/li>\n\n\n\nproxmox-host-raid1-nvme.txt<\/code><\/li>\n\n\n\nct100-openwebui-ai.txt<\/code><\/li>\n\n\n\nct101-llamacpp.txt<\/code><\/li>\n\n\n\nct102-vllm.txt<\/code><\/li>\n\n\n\nai-rig-roadmap.txt<\/code><\/li>\n\n\n\nai-rig-ideas.txt<\/code><\/li>\n<\/ul>\n\n\n\nThis made the project much more structured. The files document the host, storage design, container roles, installed models, backup rules, troubleshooting notes, and future roadmap.<\/p>\n\n\n\n
\n\n\n\nWhat I Want To Use This For<\/h2>\n\n\n\n
The immediate use case is local AI chat and technical assistance. The longer-term goal is to build a private multimodal AI workstation.<\/p>\n\n\n\n
Current and future uses include:<\/p>\n\n\n\n
- \n
- General local assistant work<\/li>\n\n\n\n
- Coding help<\/li>\n\n\n\n
- Research and summarization<\/li>\n\n\n\n
- Homelab troubleshooting<\/li>\n\n\n\n
- Documentation generation<\/li>\n\n\n\n
- Cybersecurity and IT support workflows<\/li>\n\n\n\n
- Splunk and log-analysis assistance<\/li>\n\n\n\n
- Windows Event Log review<\/li>\n\n\n\n
- Suricata alert explanation<\/li>\n\n\n\n
- PowerShell and Linux command support<\/li>\n\n\n\n
- Agent workflow testing with Hermes<\/li>\n\n\n\n
- Image-generation and thumbnail workflows<\/li>\n\n\n\n
- Speech-to-text and local voice assistant experiments<\/li>\n\n\n\n
- Local content-production workflows<\/li>\n<\/ul>\n\n\n\n
The important design rule is that I do not want to collect random models just because they exist. For each category, I want to test and keep the best-performing model or tool that my rig can realistically run.<\/p>\n\n\n\n
\n\n\n\nKey Lessons Learned<\/h2>\n\n\n\n
The biggest lessons from this project were practical, not theoretical.<\/p>\n\n\n\n
- \n
- Stable daily use matters more than theoretical performance.<\/strong>
vLLM was the most complex and interesting backend, but Ollama and llama.cpp are more useful for daily work right now.<\/li>\n\n\n\n- Total VRAM is not the whole story.<\/strong>
GPU memory, PCIe link speed, backend architecture, model format, and quantization all affect the real result.<\/li>\n\n\n\n- Consumer hardware can do serious work, but it has tradeoffs.<\/strong>
The x1 links on the RTX 3060s made some multi-GPU workloads less practical, especially vLLM pipeline parallelism.<\/li>\n\n\n\n- Bind mounts are useful but affect backups.<\/strong>
Keeping models outside the CT root disks is cleaner, but those model folders need their own backup plan.<\/li>\n\n\n\n- A service can be running before a model is ready.<\/strong>
The llama.cpp503 Loading model<\/code> response was a good example of why startup state and model readiness need to be checked separately.<\/li>\n\n\n\n- Proof-of-concept and daily-driver are different goals.<\/strong>
CT102 is successful because it proved vLLM could run the model. That does not mean it should be the main workflow.<\/li>\n\n\n\n- Documentation turns experiments into infrastructure.<\/strong>
Writing the documentation pack made the setup easier to explain, recover, and turn into a portfolio project.<\/li>\n<\/ol>\n\n\n\n
\n\n\n\nFinal Thoughts<\/h2>\n\n\n\n
This project helped me move from experimenting with local AI models to building an actual AI infrastructure foundation.<\/p>\n\n\n\n
The most important result is not just that the models run. The important result is that the system now has a structure:<\/p>\n\n\n\n
- \n
- Proxmox host documented<\/li>\n\n\n\n
- Storage layout documented<\/li>\n\n\n\n
- AI containers separated by role<\/li>\n\n\n\n
- Backends tested and compared<\/li>\n\n\n\n
- Models stored outside CT root disks<\/li>\n\n\n\n
- Warm backups created<\/li>\n\n\n\n
- Cold storage plan defined<\/li>\n\n\n\n
- Troubleshooting notes preserved<\/li>\n\n\n\n
- Future roadmap written<\/li>\n<\/ul>\n\n\n\n
For my current hardware, the best practical path is to use OpenWebUI\/Ollama and llama.cpp as the main daily-driver options, while keeping vLLM as a working proof-of-concept and learning environment.<\/p>\n\n\n\n
This gives me a strong foundation for the next phase: building a local agent workflow and turning the rig into a more capable private multimodal AI workstation.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction This project documents the first major foundation of my local AI homelab. The goal was not just to install a chatbot or download a single model. I wanted to build a self-hosted AI environment that works more like real infrastructure: separated services, documented storage, recoverable backups, and different backends for different use cases. The […]<\/p>\n","protected":false},"author":1,"featured_media":1578,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1573","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-projects"],"_links":{"self":[{"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=\/wp\/v2\/posts\/1573","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1573"}],"version-history":[{"count":9,"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=\/wp\/v2\/posts\/1573\/revisions"}],"predecessor-version":[{"id":1607,"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=\/wp\/v2\/posts\/1573\/revisions\/1607"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=\/wp\/v2\/media\/1578"}],"wp:attachment":[{"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1573"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1573"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/yairmartinezcybersecurityportfolio.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1573"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- Total VRAM is not the whole story.<\/strong>
- Stable daily use matters more than theoretical performance.<\/strong>
- llama.cpp:<\/strong> stable OpenAI-compatible backend and strong agent candidate<\/li>\n\n\n\n
- Tensor Parallel 2:<\/strong> could not fit cleanly on the two 12 GB GPUs without CPU offload.<\/li>\n\n\n\n
- Tensor Parallel 3:<\/strong> failed because model dimensions\/vocab were not divisible by 3.<\/li>\n\n\n\n