Local and On-Device AI

References

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

1. vLLM GPU worker: profiling available KV capacity

   The GPU worker profiles a model forward pass and, when configured, estimates CUDA-graph memory separately. Its available KV-cache budget subtracts profiled non-KV consumption and the applied graph estimate from requested memory. Later warmup compares actual graph-pool use with its estimate and reserves additional headroom in suggested allocations. This connects weights, activations, runtime buffers and graph storage to the space left for request state. An explicit cache-byte override skips normal capacity inference and requires a suitable value.

2. Transformers: cache allocation versus retained history

   KV caches retain attention keys and values beyond model weights. DynamicCache grows as tokens arrive; StaticCache preallocates a maximum capacity, masking unused positions and potentially wasting memory and attention work on short sequences. Sliding-window or chunked layers stop growing at their retained-history limits even when the configured sequence capacity is larger. Offloading transfers most layer caches through CPU memory; quantized caches reduce precision but can increase latency for short contexts. Operational inference: allocated slots are a storage decision, not evidence that the model supports that many sequence positions. For ordinary full-history static caching, reserve sufficient slots for prompt plus intended output while independently enforcing the model's supported context limit. Windowed cache capacity must instead follow the architecture's retention rules.

3. PyTorch Quickstart: forward computation, training, and loading

   Tensors hold shaped numerical data: a batch has dimensions for examples and features, while parameter tensors hold learned weights and biases. Forward computation applies the network's layers to input tensors using those parameters, producing prediction scores. Training additionally compares predictions with targets through a loss, calls backward() to compute parameter gradients, and calls optimizer.step() to update parameters; zero_grad() clears accumulated gradients. Loading recreates the architecture and restores its state dictionary. The example then predicts using eval() and no_grad(), without loss-driven parameter updates. Computing a loss for evaluation does not itself train the model.

4. NIST SP 500-325: Fog Computing Conceptual Model

   NIST distinguishes smart endpoints, intermediate computing nodes, and centralized cloud services. Intermediate nodes can be gateways, routers, servers, or virtual components providing computation and communication for endpoints. A cloud connection is not mandatory for every such arrangement. Placement implication: an endpoint using a nearby inference server still crosses a machine and communication boundary; proximity does not establish who owns or administers that server.

5. Why Large? Tiny LMs & Agents on Edge/Robotics

   Local inference can keep a feature available without connectivity, retain its data on-device, and avoid cloud inference charges that accumulate with interaction volume.

6. Hey Siri: An On-device DNN-powered Voice Trigger for Apple’s Personal Assistant

   Apple's October 2017 account describes a small detector continuously recognizing a wake phrase on the iPhone's low-power auxiliary processor. A sufficiently strong detection wakes the main processor for a larger model's check, avoiding continuous use of the main processor. The contemporary Apple Watch implementation instead enabled detection after a wrist raise because of its smaller battery and competing startup work. After local checks, accepted audio could reach Siri's server, which could reject a false activation. Local wake-word recognition therefore served a different workload and information boundary from the subsequent assistant request.

7. FinOps: comparable workload denominators and cost boundaries

   FinOps distinguishes resource-efficiency units, such as cost per token, from business outcomes, such as cost per resolved case. It calls for documented metric definitions, assumptions, cost inclusions and allocation rules, progressing toward fully loaded costs. Application to local-versus-remote inference: use unit cost=allocated costs over a defined period/completed useful workload units over that period. Hold task mix, quality criteria and service requirements comparable; specify input/output lengths when counting requests or tokens. Include retries and failed attempts in the cost numerator even when the denominator counts only successful tasks. Compare full ownership cost with full service-delivery cost, or explicitly label a narrower marginal-cost comparison.

8. Android: Back up user data with Auto Backup

   Android Auto Backup includes most application files, databases, and shared preferences by default, while excluding designated cache and no-backup directories. Developers can configure file inclusion and exclusion separately for supported transfer types. Consequently, storing a transcript in application-internal storage does not alone establish that it remains on that device. The documentation also warns that, on some manufacturers' devices, disabling cloud backup through allowBackup does not disable device-to-device transfer.

9. Hugging Face Hub: Environment variables

   The Hub client documents network activity beyond downloading missing weights: hf_hub_download normally checks for newer file metadata even when a file is cached, and participating libraries can send usage telemetry. Logged-in clients ordinarily send an authentication token on read requests unless implicit-token behavior is disabled. Separate controls govern Hub offline access, telemetry, and implicit authentication. The hf CLI also documents a separate PyPI update check. These distinctions provide concrete operational flows to inventory around local model execution.

10. Ollama: FAQ

    Ollama documents both local models and cloud features. It can run in local-only mode by disabling its cloud features, including cloud models and web search. Its HTTP server binds to 127.0.0.1 by default, and changing the bind address or adding a proxy can expose it on a network. Local inference therefore needs a data-flow definition: where the model executes, which services receive prompts or tool data, and who can reach the model endpoint. A locally installed application is not automatically a fully offline workflow.

11. gVisor security model: containment and host dependencies

    gVisor reduces untrusted code’s ability to exploit the host system-call interface. Its Sentry implements application system calls instead of passing them directly to Linux, and itself uses a restricted host interface; the Gofer mediates filesystem access. This adds defenses against escape, not a guarantee of immunity. Sandboxed code can still access mapped files and allowed network connections. Hardware side-channel defenses depend on the host OS and platform; resource limits depend on host cgroups, and network policy needs separate enforcement. Host-networking and directfs settings change which host operations are available. Architectural implication: a local assistant’s sandbox protects the host from untrusted execution but does not establish confidentiality against a compromised host kernel or host components controlling its isolation and resources.

12. ONNX Runtime Architecture

    ONNX Runtime loads an ONNX model into an internal computation graph, performs provider-independent optimizations, and partitions the graph into subgraphs according to available execution providers. Providers declare the operations or subgraphs they can execute through capability queries. Assigned work is compiled or handled by the provider, which also participates in memory allocation. This separates the portable model representation from hardware-specific execution. An exported file and an installed accelerator backend are not sufficient evidence that the desired operations execute on that accelerator.

13. Scikit-learn: scoring rules and baseline estimators

    Evaluation tools accept an explicit scoring rule; classification accuracy is sum_i 1[prediction_i=target_i]/n. Dummy estimators provide sanity-check baselines such as always selecting the training set's most frequent class. Accuracy can conceal poor minority-class performance, motivating metrics such as balanced accuracy. Methodological inference for technical tasks: define the correctness criterion and baseline before selection, evaluate candidates on the same validation cases, and apply the frozen criterion to the final test set. A syntax check, exact-answer comparison, and executable task test measure different properties.

14. Hugging Face model cards: locating license terms

    Hub model repositories declare licensing in the README model card's metadata. For a custom license, the documented fields are license: other, license_name and license_link; the link can point to external terms or a LICENSE file in the repository. Deployment-review inference: inspect the actual terms for the exact artifact and revision, including referenced policies and relevant upstream model terms, rather than treating the displayed tag as the complete agreement. Download availability and the inference runtime's separate software license do not establish the permissions governing model weights.

15. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

    Zechun Liu and colleagues at Meta introduced MobileLLM in a February 22, 2024 preprint. They investigated architecture choices for sub-billion-parameter models, combining deep, narrow networks, shared embeddings, and grouped-query attention; MobileLLM-LS additionally reused adjacent block weights. Their specialized API-calling experiment trained on 5,000 generated examples and tested on 2,500. MobileLLM-350M scored 65.3% intent exact match and 48.8% structure exact match, versus 62.8% and 50.9% for LLaMA-v2 7B. Separate iPhone profiling distinguished loading, initialization, and execution rather than treating startup as warm inference.

16. FunctionGemma model card

    Google reports Mobile Actions evaluation results of 58% for base FunctionGemma and 85% after task specialization, measuring identification and formatting of mobile system calls. Its separate device test used a Samsung S25 Ultra CPU, LiteRT XNNPACK, four threads, 512 prefill tokens, and 32 decode tokens. The Mobile Actions dynamic-int8 row lists a 1024-token context configuration, 288 MB model size, 551 MB peak RSS, and 0.3-second time to first token. RSS is resident set size: the reported process-memory measure differs from model-file size. The card states that FunctionGemma is not intended as a direct dialogue model.

17. Combining Neural Networks and Context-Driven Search for On-Line, Printed Handwriting Recognition in the Newton

    Apple's Print Recognizer combined a neural character classifier with searches over possible stroke groupings and words. The developers initially restricted the task to hand printing, where pen lifts helped separate strokes, while allowing words outside the dictionaries to avoid forcing every input into a known word. The authors identify December 1995 as its first shipment in Newton OS 2.0 MessagePad 120 units. Its input consisted of pen coordinates and pen-up/down information, illustrating a specialized local recognition pipeline rather than general-purpose text generation.

18. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

    Google's April 17, 2017 preprint addressed recognition tasks that needed timely execution on constrained mobile and embedded platforms. MobileNets used factorized convolutions and adjustable network width and input resolution to produce models suited to different resource limits. The paper distinguishes designing small networks directly from compressing existing networks and notes that reducing model size alone does not necessarily optimize speed. Applications included classification, detection, face attributes, and geolocation.

19. Introducing llamafile

    Mozilla announced llamafile's first release on November 29, 2023. Developed by Justine Tunney, it combined llama.cpp with Cosmopolitan Libc to package language-model weights into executable files intended to run across multiple operating systems without a separate installation. The consequential change was a distribution mechanism that brought model data and execution software together, reducing the setup required to try a local generative model.

20. llama.cpp early repository snapshot at 920a7fe

    This early llama.cpp snapshot states the goal of running four-bit LLaMA inference on a MacBook CPU using C/C++, Arm Neon, and Apple's Accelerate framework. Its README includes a published run of the 7B model and explicitly says that only that model had been tested. It also records a recently corrected bug producing invalid output. The snapshot documents a concrete accessibility contribution: a compact CPU execution path for an existing language model, with model conversion and quantization tools alongside the executable.

21. CUDA Programming Guide: Programming Model

    The host is the CPU and its memory; the device is the GPU and its memory. A kernel is a function invoked for GPU execution. Its launch organizes threads into blocks and blocks into a grid. Each block executes on one streaming multiprocessor, or SM, whose resources support computation and cooperation. A grid can contain far more blocks than can execute simultaneously, and ordinary blocks cannot assume scheduling order or depend on another block becoming resident. CUDA groups threads into 32-thread warps. Under single-instruction, multiple-thread execution, threads follow the same kernel code but may take different branches; inactive lanes are masked while a branch executes.

22. Develop AI applications for Copilot+ PCs

    A neural processing unit, or NPU, is specialized hardware for neural-network mathematical operations; software must target it to use it. Microsoft's profiling guidance separates model loading and runtime-session creation from inference, CPU preprocessing and postprocessing, individual operator timing, and NPU activity. Windows tracing can identify processes and call stacks submitting NPU work. This provides observable execution evidence beyond the presence of NPU hardware.

23. GPU delegates for LiteRT

    LiteRT delegates enable execution on specialized processors. Its GPU delegate supports a specified subset of operations and versions. Unsupported operations can leave execution split between GPU and CPU; the guide warns that synchronization overhead often makes this slower than running the whole network on CPU. Such partial delegation emits warnings rather than a runtime-failure callback. Input representation also matters: compatible four-channel camera input can avoid a conversion copy otherwise needed for three-channel images.

24. TensorFlow Lite is now LiteRT

    On September 4, 2024, Google's AI Edge team announced the TensorFlow Lite name changing to LiteRT, short for Lite Runtime. The announcement connected the change to support beyond TensorFlow, including models authored in PyTorch, JAX, and Keras. It described a progressive documentation and package transition rather than a replacement of existing applications. This links older TensorFlow Lite research and deployment examples to later LiteRT documentation.

25. ONNX Runtime: Model Usability Checker

    The mobile usability checker examines supported operations, input shapes, and the partitions assignable to NNAPI or CoreML. Its published ResNet fixture reports two CoreML partitions containing 120 of 122 nodes, with an unsupported Flatten operation and a dynamic-shape restriction. The output separately lists unchecked operator conditions. Fixing shapes changes estimated coverage but does not remove every unsupported operation. The guide requires performance testing against CPU execution rather than treating its recommendation as measured accelerator performance.

26. Deploying ONNX Runtime Web

    Browser deployment requires application JavaScript, the appropriate WebAssembly runtime binaries, and model files. Models using external tensor data need that data supplied too. Missing or incorrectly served WebAssembly binaries prevent initialization even when model files exist. Worker loading also depends on origin and content-security rules, and WebGPU requires a secure context. Offline-design implication: caching weights alone is insufficient when runtime binaries, workers, or other required assets still depend on network retrieval.

27. Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft

    The presented stack combines runtime acceleration with hardware-specific model variants and device-aware model discovery.

28. Android: Memory allocation among processes

    Android distinguishes persistent storage from RAM and zRAM, which holds compressed swapped pages within RAM. CPU and GPU access the same RAM. Resident set size, or RSS, counts a process's shared and private resident pages. Proportional set size, or PSS, assigns each process a fraction of shared pages; unique set size, or USS, counts only unshared pages. Summing RSS across processes can therefore count shared allocations repeatedly. The platform also reclaims memory and terminates processes under pressure, so total installed RAM is not a private allocation available to one inference task.

29. MLX: Unified memory

    On Apple silicon, CPU and GPU access the same memory pool. MLX arrays do not need an explicit device location; operations specify where they execute, and the scheduler handles dependencies between CPU and GPU work. This avoids the explicit array copies between separate device memory spaces found in other execution arrangements. Unified memory is an architecture property with implications for model placement, not an infinite resource or a guarantee that a given model will run quickly.

30. Linux mmap(2) manual

    Memory mapping creates a region in a process's virtual address space backed by a file or another object. Mapping is distinct from ensuring that its pages are already resident in RAM: Linux documents later page faults and a prefaulting option that performs file read-ahead to reduce subsequent blocking. Application implication: mapping a model file is not proof that its working data are resident or that execution will avoid storage access; mapped size alone is not a physical-memory measurement.

31. MCUNet: Tiny Deep Learning on IoT Devices

    Ji Lin and colleagues at MIT, National Taiwan University, and MIT-IBM Watson AI Lab presented MCUNet at NeurIPS 2020. It jointly designed constrained neural architectures through TinyNAS and model-specific execution through TinyEngine. TinyEngine generated code and planned intermediate storage using the complete model, while TinyNAS searched architectures that fit the resulting resource limits. On the default STM32F746 target, Table 2 reports 61.8% ImageNet accuracy for TinyNAS with TinyEngine versus 55.5% with CMSIS-NN under the study's runnable-model constraints. The contribution was coordinating model and runtime design rather than assuming a small weight file ensured deployment.

32. Quantize ONNX models — ONNX Runtime

    Linear quantization represents an approximate floating-point value as scale times an integer offset from a zero point. Fewer representable values reduce precision. Static activation quantization estimates parameters from calibration inputs and stores them for reuse; dynamic quantization calculates parameters during inference, adding work. Quantization-aware training retrains with quantization effects considered and is distinct from post-training conversion. The documentation recommends comparing original and quantized weights and activations to localize accuracy loss and keeping problematic tensors at higher precision when needed.

33. A Practical Guide to Efficient AI

    Weight quantization reduces the precision used to represent model weights, shrinking their storage and memory requirements.

34. Scikit-learn: separating fitting, selection, and final evaluation

    Training data fit model parameters; validation data or cross-validation compare configuration choices. A separate test set remains held out for final evaluation. Repeatedly adjusting settings to improve test results leaks test knowledge into selection and makes the reported score optimistic. In k-fold validation, each candidate trains on k-1 folds and is evaluated on the remaining fold, repeating across folds. Preprocessing must also be fitted within the training partition. Applying the same principle to local AI means selecting prompts, quantization, retrieval settings, and decoding configurations before examining final held-out task results.

35. Roofline Performance Model — NERSC

    The Roofline model relates attainable arithmetic throughput to compute capacity, memory bandwidth, and arithmetic intensity: operations performed per byte moved at a specified memory level. Low-intensity work can be limited by moving data rather than executing arithmetic. Reuse can increase intensity by doing more work per transferred byte. Roofline is an upper performance envelope; measured execution can lie below it because of additional inefficiencies. The same workload can fall in different limiting regimes on different hardware.

36. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    The illustrated inference loop has a prefill stage for the prompt followed by incremental decoding, with different computation shapes.

37. GenAI-Perf: latency and throughput measurement

    GenAI-Perf defines TTFT as first-response receipt minus request-send time, and request latency as final-response receipt minus send time. Its inter-token latency divides the interval between successive responses by the latter response's generated-token count. Output-token throughput is total generated tokens divided by benchmark duration; request throughput is final responses divided by duration. A p99 latency is the 99th percentile of the specified latency observations, not an average. The tool supports concurrency, request rate, warmup count, measurement intervals, and input/output-length controls. Reproducible comparisons must report those settings and the workload/tokenizer.

38. RIVA ASR Developer Pack Overview

    For automatic speech recognition, the guide defines real-time factor as processing time divided by audio duration. Values below one indicate processing faster than the duration of the supplied audio; the reciprocal gives the speed multiple. Its reported measurements process complete audio buffers after warmup, excluding initialization warmup transcriptions. Therefore, a favorable real-time factor for this benchmark does not by itself establish live microphone-to-text latency or the time until a streaming transcript becomes final.

39. ML Kit GenAI Speech Recognition API

    The API accepts streaming microphone or audio-file input and returns a stream of transcriptions. Early text can be partial and subject to revision before becoming final. Integration requires checking model readiness, downloading required features when absent, and closing the recognition client to release resources. This supplies a concrete workload distinction: displaying an early transcript and obtaining finalized text are different milestones.

40. Google SRE: Handling Overload

    Admission controls and per-customer quotas limit resource consumption so one workload does not exhaust shared capacity. Resource usage can be a better capacity signal than requests per second because requests vary in cost. Graceful degradation reduces work by returning less complete results or using cheaper, potentially stale cached data. Client-side throttling can prevent rejected requests from consuming backend resources. Under extreme overload, even degraded computation may be impossible and explicit errors are necessary.

41. Adding metadata to LiteRT models

    LiteRT metadata describes model inputs, required preprocessing, outputs, and associated files. Vocabulary files map text pieces to identifiers; classification labels give meaning to output categories. The documented packaging mechanism can bundle such files with the model in a ZIP-compatible .tflite artifact. Input normalization and numerical quantization are separate operations: inference must use the normalization expected by the model, while quantization handling depends on tensor types. Successfully loading weights therefore does not establish that an application supplies or interprets values correctly.

42. Transformers: model-specific chat serialization

    A chat template converts ordered role/content messages into the token sequence a causal model continues, inserting model-specific role markers, message boundaries and special tokens. Models fine-tuned from the same base can require different formats: Mistral-Instruct brackets user messages with instruction delimiters, while Zephyr uses explicit speaker markers. The guide warns that incompatible control tokens degrade performance. Where the format requires it, add_generation_prompt=True appends the assistant-start marker; omitting it can cause continuation of the user's message instead of a reply. Some templates need no such marker. Templates already supply required special tokens: use apply_chat_template(tokenize=True), or tokenize the rendered string with add_special_tokens=False to avoid duplicated beginning/end tokens.

43. Core ML: Downloading and Compiling a Model on the User’s Device

    Apple documents downloading models after application installation as an alternative to bundling every model, allowing applications to fetch only the models needed for particular features or locales. The downloaded .mlmodel must be compiled into .mlmodelc before this workflow creates an MLModel instance. Compilation produces a temporary artifact; Apple recommends moving reusable compiled models into persistent application storage to avoid repeating download and compilation. Delivery, preparation, and persistent readiness are therefore distinct stages.

44. Understand built-in model management in Chrome

    Chrome documents resumable model downloads after connectivity loss, continuation after the initiating tab closes, and browser-restart resumption within its stated conditions. Some APIs require additional adaptation weights alongside the base model. Updates download a complete replacement while the existing model remains usable; a prompt running during the switch can fail. Disk pressure or eligibility changes can purge the model, including during a session. After deletion, an application must trigger a new download. Availability at session start therefore does not establish continued offline readiness.

45. Android AtomicFile

    AtomicFile writes replacement data into a new file, synchronizes completed data to disk, and then renames it over the original. finishWrite commits the replacement; failWrite closes the stream and removes the new file. The class supplies no locking, so callers must prevent conflicting accesses. A constructed model-installation design could use this primitive to replace a small active-version record after staging and validating its referenced artifacts; the documented guarantee concerns one file, not the whole model package.

46. The Update Framework: authenticated update workflow

    TUF starts from trusted root metadata specifying authorized keys and signature thresholds. Clients verify signed timestamp, snapshot, and targets metadata, then verify downloaded targets against authenticated hashes and lengths. Signatures and target hashes resist unauthorized substitution. Persisted metadata versions allow rejection of older metadata, resisting rollback; cross-role version checks also prevent inconsistent combinations. Expiration checks against the fixed update-start time detect attempts to keep clients indefinitely on stale metadata. These mechanisms complement one another: a valid signature alone does not prove freshness.

47. LiteRT: Benchmark Interpreter API

    LiteRT's benchmark tooling separately measures initialization time, warmup inference, steady-state inference, initialization memory usage, and overall memory usage. Android and iOS benchmark applications share core measurement logic with command-line tools, while options and output differ by environment. The guide cautions that even its Android benchmark application's numbers differ from inference integrated into the actual application.

48. Chrome Prompt API: stopping prompts and managing sessions

    Chrome's Prompt API accepts an AbortSignal for stopping an individual prompt. Destroying a session aborts ongoing execution, frees its resources, and makes the session unusable. The guide notes that keeping a session can avoid repeated creation costs. It also demonstrates rebuilding conversational context through initialPrompts after a browser restart. This is restoration from supplied messages, not a documented checkpoint of an interrupted model computation.

49. Android: Processes and app lifecycle

    Android controls application-process lifetime using component activity, user importance, and available system memory. Its documented example shows a thread started by a broadcast receiver being terminated when the receiver returns and the process becomes reclaimable. Processes are prioritized for termination under memory pressure; even foreground processes can be killed as a last resort. Application implication: an active inference thread and its in-memory state do not constitute durable task progress.

50. Chrome: Page Lifecycle API

    A frozen page suspends freezable tasks such as JavaScript timers and fetch callbacks. A discarded page is unloaded to conserve resources and cannot run a final cleanup callback. Chrome recommends saving state before these transitions and checking document.wasDiscarded when a page is loaded again. Application implication: browser inference cannot rely on a visible tab, an outstanding promise, or an unload handler to preserve unfinished work; recovery must reconstruct the task from state that was actually saved.

51. Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft

    Download the required model before losing connectivity; local execution does not imply offline first-time setup.

52. ONNX Runtime Java API: OrtSession.RunOptions

    Setting RunOptions.setTerminate(true) requests termination as soon as possible for all incomplete run calls using that RunOptions instance. Resetting the flag allows the options to be reused. Application implication: sharing the same options object across concurrent requests also shares the cancellation scope; request-specific cancellation requires controlling which calls use that object.

53. Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency

    Dynamic voltage and frequency scaling, or DVFS, changes component operating settings to balance power and execution time. This study measures device energy and inference latency on Pixel 7 and Pixel 7 Pro using llama.cpp with GPU execution. It compares default CPU, GPU, and memory governors with controlled frequency combinations, finding combinations that reduce latency without increasing energy for tested workloads. Its energy accounting incorporates both power and runtime: lower instantaneous power does not necessarily mean less energy per completed workload.

54. Android Thermal API: sustained performance limits

    Android documents that thermal state depends on environmental conditions, recent use, and device thermal design. High performance may be sustainable only temporarily before throttling; an overheated device may need workload below its sustainable level to dissipate heat. Thermal status and headroom help identify this constraint. Inference-testing implication: compare initial performance with repeated measurements during prolonged identical load, recording elapsed time, thermal observations, latency, throughput, and environmental conditions. A decline coinciding with thermal limits supports a thermal explanation more strongly than a single short benchmark.

55. Android Power Profiler: observations and attribution limits

    Android Studio's Power Profiler exposes system-trace observations from the On Device Power Rails Monitor, separating available hardware subsystems such as CPU, GPU, memory and display. The guide specifies Pixel 6 and later Pixel devices running Android 10/API 29 or higher; available rails vary by device. Unsupported devices may expose battery capacity in percent, remaining charge in microampere-hours and instantaneous current in microamperes through battery gauges. ODPM observations are device-level, not app-specific, so other active applications introduce noise. The guide supports correlating traces with app activity and comparing alternative implementations. Inference-testing application: repeat an identical sustained workload under controlled background activity, recording energy alongside completed work, latency, throughput and thermal state.

56. Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind

    Distinguish occasional interactive inference from sustained processing, and defer latency-insensitive batches to charging periods.

57. Overview of the ML Kit GenAI APIs

    ML Kit describes GenAI inference inputs and outputs as processed on-device through AICore, with a model shared across applications. Device and language availability vary by API and downloaded model. Different Gemini Nano versions can produce different outputs for the same prompt; getBaseModelName() exposes the version identity. AICore applies short-period request quotas and longer-duration battery-use quotas. GenAI inference requires the top foreground application; even a foreground service can receive BACKGROUND_USE_BLOCKED.

58. MELTing point: Mobile Evaluation of Language Transformers

    MELT ran four-bit Zephyr-3B on an iPhone 14 Pro through 50 consecutive prompts, repeating the experiment three times, and observed declining throughput. The authors hypothesize changes in energy and frequency-management modes rather than demonstrating those transitions directly. Their infrastructure uses battery-bypass power monitoring, thermal imaging, and synchronized application events; communication baselines are subtracted. A separate Instruments trace of Zephyr-3B shows memory allocation and GPU computation occurring sequentially, with compute waiting during allocations. These observations connect sustained performance and execution traces to device feasibility.

59. Transformers installation: offline mode

    Transformers requires necessary files to be downloaded and cached before use in an offline or firewalled environment. The guide demonstrates downloading a model repository in advance. HF_HUB_OFFLINE=1 prevents HTTP calls to the Hub during model loading, while local_files_only=True restricts from_pretrained() to local files. A populated cache and an explicit offline loading policy are distinct from merely having the Python library installed.

60. TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM

    Offline operation depends on the selected skill's dependencies, even when model inference runs locally.

61. Android: Build an offline-first app

    Android distinguishes online-only writes, queued writes transmitted after connectivity returns, and lazy writes persisted locally before synchronization. Lazy writes can create conflicts requiring reconciliation. Its offline to-do example preserves newly created tasks locally; that local success precedes network synchronization. Pull-based caches can become stale or lack data during prolonged disconnection. The guidance also says unauthorized requests should not be retried until suitable credentials are available. Application implication: locally generated output, locally saved work, and a completed remote action are separate outcomes.

62. Temporal Activity Execution

    An Activity Execution can comprise multiple task attempts. Temporal relies on timeouts to detect lost work, including worker crashes after invocation, and retries according to policy; limiting attempts to one prevents retry but does not prove an external effect failed. Cancellation is cooperative: activities receive service cancellation through heartbeats, can ignore it, and workflows may proceed without waiting for acceptance. A timed-out attempt may therefore continue while another attempt runs. Application consequence: treat an unconfirmed external mutation as uncertain, retain its operation identifier, reconcile against the receiving system, and use enforced idempotency or explicit recovery before repeating it. Timeout or cancellation is not evidence that a payment, message, or write was reversed.

63. Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge

    The University of Michigan's 2017 Neurosurgeon study investigated splitting neural-network execution between a mobile platform and a server because transferring inputs could outweigh faster server computation. Its AlexNet comparison used a 152 KB image, a Jetson TK1, and a Tesla K40 server with Caffe. Local GPU execution consumed less mobile-device energy than cloud processing over the studied LTE and 3G connections, while cloud processing over Wi-Fi consumed less than local execution. Layer-level partitioning could transfer intermediate results instead of the original input; the selected boundary depended on network conditions and server load.

64. Compilers in the Age of LLMs

    Hybrid inference is presented as a future architecture in which small models near users cooperate with larger cloud models.

65. W3C Media Capture and Streams

    The specification defines microphone and camera permissions as access to capture devices. A granted permission does not guarantee capture succeeds, because other constraints can prevent it. Its security discussion warns against using stored permission to automatically transmit media to a destination selected by a third party. Application implication: microphone access and authorization of a particular outbound destination are separate decisions; a capture permission should not serve as the application's cloud-transfer policy.

66. Text Embeddings Reveal (Almost) As Much As Text

    Embedding inversion attempts to reconstruct text from its numerical embedding. The authors demonstrate reconstruction using a trained inversion model that repeatedly proposes text, re-embeds it, and corrects the proposal. Their threat model gives the attacker the target embedding and query access to the embedding model. The experiments also recover names from embedded clinical notes. Thus converting text into vectors locally does not by itself establish that transferring those vectors conceals the original information.

67. Firebase AI Logic: hybrid inference on Android

    Firebase distinguishes unavailable, downloadable, downloading, and available on-device model states. An inference request does not automatically download a missing model: it falls back or fails according to the configured mode. warmup() separately loads the model and initializes runtime components. PREFER_ON_DEVICE permits cloud fallback; ONLY_ON_DEVICE throws when local execution cannot proceed. Unsupported request features can also trigger permitted fallback. Responses identify whether execution was local or cloud-hosted.

68. Private Cloud Compute: A new frontier for AI privacy in the cloud

    Apple's 2024 Private Cloud Compute description makes the remote boundary explicit: the device constructs a request containing a prompt, selected model, and inference parameters, then encrypts it to validated processing nodes. Those nodes must access request data to compute a response; supporting load balancers do not receive the decryption keys. Apple describes deleting user data after returning the response and restricting operational metrics rather than exposing general-purpose inspection tools. This is remote processing with specified protections, not computation confined to the user's device.

69. Stable Diffusion with Core ML on Apple Silicon

    Apple's implementation distinguishes first-load model preparation from subsequent cached loads. Its FAQ says preparation depends on the selected compute unit, and that using compiled .mlmodelc assets enables caching that .mlpackage loading does not provide in this pipeline. The mobile deployment guidance recommends testing the lowest-memory supported target, documents runtime peaks exceeding 2 GB depending on compute units, and offers a reduceMemory option. It warns that system load, model version, compute selection, and application design affect whether the process remains within its memory limit.

70. Playwright: Service Workers

    Service workers can mediate browser requests and provide cached offline responses. Playwright's published fixture distinguishes a page's request from the service worker's actual network request: both can produce events, but only the latter is routable when the worker handles the page request. Disabling service workers can simplify interception while changing application behavior. The guide also documents that requests for updated service-worker main scripts cannot currently be routed. Browser-level request observations therefore need a stated coverage boundary when qualifying offline operation.

71. MLPerf Tiny Benchmark

    Colby Banbury, Vijay Janapa Reddi, and collaborators presented MLPerf Tiny at NeurIPS 2021 to make heterogeneous embedded inference systems comparable. Version 0.5 paired keyword spotting, person-presence detection, image classification, and anomaly detection with datasets and quality targets. Its closed division constrained models and evaluation while allowing implementation changes. Timing excluded preprocessing and postprocessing. Energy measurement divided energy consumed during the timing window by completed inferences, reporting microjoules per inference. The reference implementation used TFLM on a NUCLEO-L4R5ZI board.

72. MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

    MobileAIBench separates desktop/server task evaluation from an iOS application measuring mobile latency and resource utilization. Its tasks include summarization on CNN/Daily Mail and XSum, scored with ROUGE measures. Mobile measurements include time to first token, total time, CPU and RAM usage, and battery drain; the reported device experiments use an iPhone 14. The framework illustrates why task effectiveness and device feasibility require different observations.

73. Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

    An agent can run its core inference locally while calling external knowledge APIs; offline operation and privacy depend on the tools it invokes.

74. From model weights to API endpoint with TensorRT-LLM

    Batch-size effects can have sharp latency changes and eventually encounter GPU-memory limits.

75. Infra behind Krea 2 - How to train and serve at scale

    Krea treated overheated GPUs as replacement candidates rather than spending training time troubleshooting them in place.

76. Frontier AI at Home (literally)

    The speaker prioritizes memory capacity, memory bandwidth, and energy per byte moved for low-batch local decode.

77. A Practical Guide to Efficient AI

    Evaluate task quality and trust and safety alongside device latency, hardware usage, and battery drain; Mobile AI Bench is presented as tooling for this work.

78. Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind

    Local inference processes prompts without sending them to a server; the described AICore service also isolates requests and does not retain their inputs and outputs.

79. Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind

    The described hybrid approach routes inference locally when Gemini Nano is available and otherwise uses cloud inference.

80. Announcing TensorFlow Lite

    The TensorFlow team released TensorFlow Lite's developer preview on November 14, 2017, targeting smaller runtime binaries, faster initialization, and mobile acceleration. Its architecture separated model conversion, a FlatBuffers model file, application APIs, and an interpreter executing supported operators. Initial examples included MobileNet image classification and Smart Reply. TensorFlow Mobile remained the production option while Lite matured; the preview deliberately supported a constrained model set.