oMLX is a local model inference server for Apple Silicon Macs. On top of MLX inference, it adds continuous batching, a two-tier RAM and SSD KV cache, automatic model loading and eviction, an admin dashboard, and OpenAI- and Anthropic-compatible APIs.
Source: github.com/jundot/omlx View on GitHub →It addresses two common problems for local coding assistants: repeated prefill makes long conversations slow, while keeping several models resident can exhaust unified memory. The official implementation reuses context through prefix sharing and an SSD cold cache, and manages models with LRU, TTL, and a total memory limit.
Source: github.com/jundot/omlxConcurrent requests are scheduled by mlx-lm BatchGenerator. Matching prefixes can be reused between a hot RAM tier and a cold SSD tier, with cold cache entries stored as safetensors.
Source: github.com/jundot/omlxOne service can discover LLM, VLM, OCR, embedding, and reranker models, while LRU, pinning, and per-model TTL settings control memory use.
Source: github.com/jundot/omlxThe server exposes OpenAI-style APIs, the Anthropic Messages API, and a local admin dashboard. A non-loopback listener requires an API key.
Source: github.com/jundot/omlxThe official DMG includes a native Swift and SwiftUI menu-bar application plus a CLI shim. Homebrew and source installations can also manage the service with omlx start, stop, restart, and serve.
Source: github.com/jundot/omlxA FastAPI layer exposes the OpenAI and Anthropic APIs. EnginePool manages text, vision, vector, and reranking engines. The scheduler uses mlx-lm BatchGenerator. PagedCacheManager, the RAM hot cache, and the SSD cold cache form the cache stack, while ProcessMemoryEnforcer applies the total memory limit and TTL.
Source: github.com/jundot/omlxPython >=3.11,<3.14macOS 15.0+Apple Silicon M1-M5oMLX suits Apple Silicon users who want a local backend for coding assistants, offline chat, vision and OCR, embeddings, or reranking, with multiple models and memory use managed from one dashboard.
Source: github.com/jundot/omlxThe latest stable version is v0.6.4, published on 2026-08-29; the latest prerelease is v0.7.0.dev4, published on 2026-09-18. These release lines must not be conflated.
Source: github.com/jundot/omlx/releasesstatus: docs-only | checked_at: 2026-09-23 | environment: The official README, pyproject.toml, Apache-2.0 LICENSE, GitHub Releases, and repository metadata were reviewed; the project was not installed or run on an Apple Silicon Mac. | limitations: No local installation or runtime verification was performed. Model loading, throughput, SSD-cache hits, memory release, custom kernels, API compatibility, remote authentication, and multi-Mac clusters were not tested. | observed_output: The official latest API returns stable v0.6.4 dated 2026-08-29. Releases also list prerelease v0.7.0.dev4 dated 2026-09-18. pyproject.toml requires Python >=3.11,<3.14.
Source: github.com/jundot/omlx/releases/tag/v0.6.4 (1) · github.com/jundot/omlx/blob/main/pyproject.toml (2)MLX LM also provides a local HTTP server, whose documentation says it performs only basic security checks and is not recommended for direct production use. oMLX reuses the mlx-lm BatchGenerator and adds a multi-model pool, SSD cold cache, admin UI, menu-bar app, and API-key enforcement for non-local listeners.
Source: github.com/jundot/omlx (1) · github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md (2)If you have an Apple Silicon Mac and want to run several local models, reuse long conversation prefixes, and expose one backend to coding tools, oMLX is worth a small localhost trial. Before production or LAN use, test model compatibility, cache size, and authentication. For one simple model service, the smaller MLX LM server may be enough.
Source: Official releasesNot through the officially supported path. The current requirements are Apple Silicon, macOS 15.0 or later, and Python 3.11 through 3.13.
Choose 0.6.4 for a stable environment. GitHub marks 0.7.0.dev4 as a prerelease, and its Release notes advise returning to an earlier development build if a regression appears.
Configure an API key before changing the listening address. The official documentation says the server refuses to start on a non-loopback address without an API key.
No. It can reuse a cache entry only when the request prefix matches. Disk speed, cache limits, model changes, and context changes all affect the result. This site has not measured a specific speedup.