08:53
85d ago
r/LocalLLaMA· rssEN08:53 · 05·04
→A Basic LLM Litmus Test: Python Code to Sort C: Drive Folders by Size
Reddit user KptEmreU shared one LLM code test: write Python to scan C: and sort folders by size. They say local models failed, with double-counted file sizes and nested recursive functions. The post does not disclose model names, runtime setup, or logs.
#Code#Benchmarking#KptEmreU#LocalLLaMA
editor take
A Redditor claims local LLMs all fail a simple "scan C: and sort folders by size" code test — but doesn't name which models.
sharp
KptEmreU tested local models with one filesystem script task, and only the title plus summary are available; model names, setup, prompt, code, and logs are undisclosed.
I don’t buy this as evidence that local LLMs fail at coding. It is a useful smoke test, not a benchmark. The task is simple on paper: write Python that scans Windows C: and returns folders sorted by size. The summary names two concrete failures: double-counting file sizes and nesting a recursive function inside another recursive function. That is enough to raise an eyebrow. It is not enough to indict a model family.
The missing details matter here. We do not know whether the user tested Qwen, DeepSeek Coder, Llama, Mistral, Codestral, or a heavily quantized 7B model. We do not know whether the prompt asked for permission handling, symlink handling, or avoiding double counts. We do not know whether the failure was syntax, logic, permissions, Windows paths, or runtime behavior. Reddit returning 403 means the actual post body is unavailable, so the current evidence is a title and a secondhand summary.
Still, I get why LocalLLaMA users reacted to this. Filesystem traversal is a deceptively good code-model test. It is not a pure-function HumanEval problem. It forces the model to juggle os.walk or pathlib, PermissionError, FileNotFoundError, directory aggregation, sorting, Windows drive semantics, junctions, symlinks, and duplicate accounting. A human junior developer sees a boring utility script. A model sees a bag of patterns, and that is where the failure mode shows up.
The double-counting issue is especially diagnostic. There are two valid strategies. One is bottom-up traversal, computing each directory’s own files and adding child totals once. Another is scanning every file once and propagating its size to each parent directory. Bad generated code often blends both approaches. It sums each folder, then recursively adds subfolder totals again. The output looks plausible until you test a nested tree. That is exactly the kind of bug leaderboard-style code tasks miss.
This is also where “local model” is too broad a label. Open-source code models have moved far beyond toy completions. DeepSeek-Coder-V2, Qwen2.5-Coder, Codestral, and later coder-tuned variants have been genuinely useful on standard coding tasks. But an 8B 4-bit model without execution feedback will fail this sort of dirty-environment script more often than Claude Sonnet or GPT-4.1-class systems. The gap is not just syntax quality. The gap is boundary-condition paranoia.
A strong answer should do several boring things. It should keep the script read-only. It should avoid following symlinks by default. It should catch PermissionError and FileNotFoundError. It should decide whether folder size means direct files only or recursive total. It should say that scanning C: can take time and requires permissions. It should write results to stdout or a file, not mutate the disk. If a model does none of that, I would not trust it inside an agent loop.
That agent angle is the practical reason this tiny Reddit post matters. Agents rarely spend their day solving LeetCode. They read directories, inspect repos, move files, parse logs, run scripts, and patch stateful systems. If a model double-counts a directory tree, the next agent step can make a worse decision: delete the wrong cache, compress the wrong folder, or report fake disk usage. The bug is small. The workflow risk is not.
My pushback is aimed at the post framing. A single undocumented test cannot support a sweeping claim. The title discloses one task. The summary discloses two failure types. The body does not disclose model list, parameter sizes, quantization, sampling settings, system prompt, generated code, runtime, or expected output. Without those, this is a plausible complaint, not reproducible evidence.
I would keep the test and formalize it. Build a temporary directory fixture with three levels, duplicate filenames, an empty folder, a simulated permission failure, and one symlink. Define expected recursive sizes with pathlib. Ask each model for a script, run it under pytest, and score correctness plus safety behavior. That would separate “cannot write Python” from “misses Windows filesystem edge cases.”
So my take is narrow: don’t cite this Reddit item as proof that local models are bad. Do use it as a reminder that coding benchmarks still overrate models when they stay inside pure functions. Real automation lives in messy state, and many models still lose their footing there.
HKR breakdown
hook ✓knowledge ✓resonance ✓
52
SCORE
H1·K1·R1